3D Equivariant Visuomotor Policy Learning via Spherical Projection
Authors: Boce Hu
Dian Wang
David Klee
Heng Tian
Xupeng Zhu
Haojie Huang
Robert Platt
Robin Walters
Venue: NeurIPS 2025 spotlight ()
Keywords: Equivariance
Visuomotor Policy Learning
Robotic Manipulation
Primary area: deep_learning
TL;DR: We introduce Image-to-Sphere Policy (ISP), a novel SO(3)-equivariant policy learning framework with single eye-in-hand RGB inputs.

Abstract: Equivariant models have recently been shown to improve the data efficiency of diffusion policy by a significant margin. However, prior work that explored this direction focused primarily on point cloud inputs generated by multiple cameras fixed in the workspace. This type of point cloud input is not compatible with the now-common setting where the primary input modality is an eye-in-hand RGB camera like a GoPro. This paper closes this gap by incorporating into the diffusion policy model a process that projects features from the 2D RGB camera image onto a sphere. This enables us to reason about symmetries in $\mathrm{SO}(3)$ without explicitly reconstructing a point cloud. We perform extensive experiments in both simulation and the real world that demonstrate that our method consistently outperforms strong baselines in terms of both performance and sample efficiency. Our work, $\textbf{Image-to-Sphere Policy}$ ($\textbf{ISP}$), is the first $\mathrm{SO}(3)$-equivariant policy learning framework for robotic manipulation that works using only monocular RGB inputs.

3D Equivariant Visuomotor Policy Learning via Spherical Projection
Boce Hu'
Dian Wang 2
David Klee
Heng Tian
Xupeng Zhu Haojie Huang'
Robert Platt
Robin Waltersi.
Northeastern University
Stanford University
Fqqual Advising
https://isp-2a: gdtpppu....tttbioio..
Abstract
Equivariant models have recently been shown to improve the data efficiency of diffusion policy by significant margin. However, prior work that explored this direction focused primarily on point cloud inputs generated by multiple cameras fixed in the workspace. This type of point cloud input is not compatible with the now-common setting where the primary input modality is an eye-in-hand RGB camera like a GoPro. This paper closes this gap by incorporating into the diffusion policy model a process that projects features from the 2D RGB camera image onto sphere. This enables us reason about symmetries in (3(3) without explicitly reconstructing point cloud. We perform extensive experiments in both simulation and the real world that demonstrate that our method consistently outperforms strong baselines in terms of both performance and sample efficiency. Our work, Image-to- Sphere Policy (ISP), the first (3))))-uuvvarint policy learning framework for robotic manipulation that works using only monocular RGB inputs.
Introduction
The eye-in-hand configuration, where the pri- mary perception modality is camera mounted near the wrist of the robot, is an important set- ting for robotic policy learning. This setup avoids the need for carefully calibrated external camera systems, is easier to integrate onto mo- bile robot platforms, and provides fine-grained visual details in the regions where the end- effector interacts with the environment. More- over, it is used in a growing number of large robot datasets [27, 28, 42, 29, 46].
Figure 1: We propose the first (0(3)) equivariant policy learning framework based on single eye- in-hand RGB image, where the predicted action sequence transforms equivariantly under the same group action E SO(3) applied to the whole scene. network architectures for leveraging equivariant structure this setting using only RGB input. Equiv- ariant neural networks improve data efficiency and generalization by incorporating prior knowledge of domain symmetries directly into the model [34, 60, 41], and have recently been shown enhance the performance of diffusion policy [1, 70]. However, existing equivariant diffusion policy frameworks perform best with point cloud data captured from multiple depth cameras [58]. When used with RGB data, current equivariant policies are unable to fully leverage the SO(3) structure present in the problem and underperform the point cloud version significantly [65]. This naturally raises the question: can -quivvaanaee be achieved directly from monocular RGB images to support
Despite recent advances in equivariant learn- ing [66, 63], there remains a lack of effective
Correspudingg to di aassagdstanfford.cd
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

data-efficient visuomotor learning? Such capability should also have the potential to serve as modular, plug-and-play component that generalizes seamlessly richer sensing setups.
This paper addresses this challenge by introducing a novel diffusion policy framework that incorpo- rates (0 extracted from 2D RGB observations onto a sphere and then rotates the resulting spherical signal to compensate for camera motion. This yields a stable, -quuivaauia representation that is well-suited for downstream equivariant architectures. Unlike prior work that relies on segmented point clouds [58. 70] or calibrated multi-camert systems [65], our approach maintains equivari- ance throughout the entire policy and supports robust, sample-efficient closed-loop control directly from raw eye-in-hand inputs. To the best of our knowledge, this the first framework learn (3(3) (3) equivariant visuomotor policies from monocular RGB observations in eye-in-hand settings.
into eye-in-hand visuomotor learning. Our method first projects features
Our key contributions are summarized as follows:
We introduce Image-to-Sphere Policy (ISP), the first -equivvrva policy learning frame- work that uses spherical projection from 2D RGB inputs to model 3D symmetries.
We theoretically prove that our method achieves global (0(3))-quivarinn) and local SO(2)- invariance, facilitating policy learning.
We validate our method through extensive experiments, achieving average success rate im- provement of 1 11.6% over twelve simulation tasks and 42.5% across four real-world tasks.
2 Related Work
Eye-in-hand Policy Learning Eye-in-hand policy learning [22, 26, 69, 67, 11, 43] has become a flexible and scalable alternative traditional systems that rely on multiple fixed, externally mounted cameras with precise calibration [65, 25, 58, 71. 5]. By mounting cameras on the robot's wrist, these methods simplify deployment, avoid explicit calibration, and ease demonstration collection [6, 36, 52, 233. However, the constantly shifting viewpoint introduces challenges like partial observability, which motivates the use of closed-loop policies that can handle local, viewpoint-dependent observations [72, 23, 4, 47, 73]. Recent work has explored transformer-basee [59] or diffusion- -bsed [15] architectures for eye-in-hham -a--in-han-ccontro [74, 11], showing promising results across diverse manipulation tasks. Despite this progress, existing approaches often require large-scale demonstration data [35, 37], and often overlook symmetry structures inherently present in observations. Our method addresses this gap by introducing equivariant representations that encode geometric structure for -n--in settings.
Closed-loop Visuomotor Policy Learning Early approaches closed-loop visuomotor policy learning relied on reinforcement learning and CNN-based policies to map visual inputs to single-step actions [31, 72, 27]. While effective in simple tasks, these methods were sample-inefficient and struggled to capture multimodal behaviors, as each action was predicted independently without considering temporal context. To address this, subsequent work introduced temporal modeling into behavior cloming frameworks, such as BCRNN [39] and ВеТ [53], to improve sequential consistency and planning horizons. Building on this direction, recent advances have adopted generative policy models [5, 44, 71, 65], which model multi-step action sequences as a denoising process conditioned on observations. These approaches offer stronger expressiveness and improved multimodal behavior modeling. ISP extends this line of work by further integrating structural inductive biases, which enable more generalizable closed-loop control in complex manipulation settings.
Equivariance in Robotic Manipulation Equivariant and invariant representations have been shown to improve performance and sample efficiency [13, 54, 10, 56, 33, 18, 14, 32, 7]. Prior work has in- corporated equivariant architectures for open-loop pick-and-place tasks [76, 64, 17, 19, 9, 45, 61, 21], showing strong performance with fewer demonstrations. Recently, equivariance has been extended to architecture to enhance Diffusion Policy [5]. EquiBot [70] adopts an (IM structure, and ET- SEED [58] performs trajectory-level
closed-loop diffusion policies [65, 70, 58]. EquiDiff [65] employs an
diffusion, both leveraging segmented point cloud inputs model spatial symmetries, thereby improving policy generalization. However, these approaches typically rely on multi multi-camera setups with fixed viewpoints preprocessed 3D inputs. These constraints reduce their practicality in eye-in-hand settings, where the continuously shifting viewpoint and monocular RGB input violate the assumptions of existing equivariant models. To fill this gap, ISP models symmetry in the eye-in-hand RGB setting, preserves (00))--qquvariincce and can integrate with other frameworks to enhance their effectiveness without additional preprocessing.

---

Background
3.1 Representations of (3)3)
A group representation encodes symmetry by mapping elements of a group to linear transformations. In this work, we focus on the special orthogonal group (O(3) of 3D rotations. A representation of SO(3) is homomorphism p: SO(3) + GL(V), where V is a finte-dimennional vector space and GV. denotes the group invertible linear transformations on V. We highlight three commonly used representations in robotics and geometric deep learning:
Degree-O trivial representation Po: Maps every g SO(3) to the identity transformation IR. This is used for rotation-invarian quantities, such as scalar sensor readings or gripper states. Degree-1 standard representation P1: Maps (O(3) to 3 x 3 rotation matrix acting on IR3 via (g) = gu, capturing vector features like positions and directions.
Higher degree irreducible representations pe: For E N, the representation pe: SO(3) GI,(R is given by the Wigner D-matrix of degree f. is used to describe higher degree features, such as relative poses, and is often used for latent features in equivariant neural networks.
3.2 Spherical Harmonics and Fourier Coefficients
Spherical harmonies y S + R form an orthonormal basis for square-integrabe functions on the 2-sphere and realize the irreducible representations of SO(3). Any spherical function s2 Rd can thus be expanded as:
Φ 6) = Σ
(0,
t=0 m=-
where of are the corresponding Fourier coefficients. The mapping + + {c7" is known as the Spherical Fourier Transform. Under a rotation R E SO(3), each coefficient vector ce E 2 transforms linearly via the representation pe:
(e cc.
This enables efficient otaaion-cquivarian operations on spherical signals in the spectral domain.
3.3 Diffusion Policy
Diffusion-based policy learning [24, 48] is class of imitation learning methods that model distribu- tions over action trajectories using denoising diffusion probabilistic models (DDPMs) [15]. These methods iteratively denoise sequences of noisy actions, conditioned on observations, to recover expert-like behavior. Formally, given an observation O and diffusion timestep the policy predicts a noise estimate ex from a corrupted action sequence a = a + ex using denoiser network r. The model is trained minimize the denoising objective Laitt Eg k,c [||'([, a², k) e I*1|21 [| At test time, the policy generates actions by iteratively denoising a randomly initialized sequence from Gaussian noise. Recent extensions [65] incorporate symmetry priors by designing the denoiser to be equivariant with respect to a transformation group G. Specifically, for compact groups such as SO(3), the denoiser I' is required satisfy the equivariance constraint:
T[g O, g a k) = g I(O, a²,
Aig G.
(3)
This formulation ensures that the denoising process respects the symmetry the environment.
3.4 Problem formulation
We study closed-loop robotic visuomotor policy learning through behavior cloning, where a policy trained imitate expert demonstrations. Given an observation sequence O {ot=k+1, o, at where & and n are the timestep f, the learned policy predicts an action chunk A = = /a,+1:.., ar }.
observation length and prediction horizon, respectively. Each observation o = (I, P) consists of an RGB image from the wrist camera, proprioceptive input P describing the end-effector pose. Prior work has shown that higher performance achieved when using absolute action representations, i... actions expressed in the world frame [5, 65]. Following this, we represent each predicted action

---

Robot Propriacaption
Equi Linear
Fourer
P
Driperr Orientation
Equi-Encoder
amago Sphere Projection
R2
Spherical
(4)
Ln p.R.)((a)
Equivariance Correction
(a) SO(3) Equivariant Observation Encoder
Fourier
Naley Action Sequence
A (#143......
[
4
Diflasion Step
1D Equivariant Temporal Unet
SO(3) Equivariant
Figure Overview of Image-to-Sphere Policy (ISP) (a) An 50(3)-equivarian) observation encoder extracts features from the RGB input, projects them onto the sphere, and applies an equivariance correction using the gripper orientation R. to account for the camera's dynamic viewpoint (red arrow). The corrected spherical signal Poom (r) is then processed by spherical convolution layers to extract SO(3) signals. Proprioceptive inputs are embedded via equivariant linear layers. Both image and proprioceptive features are represented as set of Fourier coefficients on (O(3) and fused (yellow block). (b) The encoded spherical signals are transformed back the spatial domain via inverse Fourier transform, sampling finite group elements as the conditioning vector for SO(3) -3)--vvvraariant denoising. The noisy action sequence is processed the same way, through equivariant linear layers and projected onto the same group elements.
E IR10 the absolute end-effector pose including a position in IR 3, an orientation represented as 6D rotation vector R (see [75]), and a gripper open state in R1. As noted by [65], the absolute action parametrization has symmetry: 3D transformations of the world frame result in the same 3D transformations to the action. We formalize the equivariance properties of ISP in Section 4.1.
Method
Figure 2 illustrates overview of our proposed method, which consists of two key components, an (0(3)-qquivarian observation encoder (Figure 2a) and an (eeuiqquuiaari diffusion module spherical projection to map image-extracted features
(Figure 2b). The observation encoder onto a hemisphere and applies spherical convolutions to ensure -euuiiqquvvrriance, producing the conditioning vector for the diffusion process. The diffusion module is designed as an SO(3)- cquivariant function of the conditioning vectors and noisy inputs. As result, the entire policy is end-to-end SO (3)-qquuivrri In the following subsections, first describe our observation encoder, which extracts SO(3) -equivariant features from 2D images, and then our equivariant diffusion module.
4.1 SO(3) Equivariant Observation Encoder
This section describes how construct an (Qeui-equivariant observation encoder that maps 2D images and robot proprioception into a 3D feature representation. The observation x E X consists of two parts, an eye-in-hand RGB image I. that captures visual information, and proprioceptive data, P E R², including the end-effector's 6D pose (position and orientation) and gripper state. Both these signals need be represented in way that encodes equivariance. Representing / is relatively easy. Following [65, 70, 51], we embed end-effector pose in SO(3) using the standard representation and gripper state using the trivial representation (Section 3.1). contrast, encoding the 2D image input into (0)33)equivarial
features is harder because changes in the pose of the induce out-of-plane viewpoint variations that are hard to model. We address this by projecting a standard 2D encoding of the image onto the sphere, described below and first proposed in the context object pose estimation [30, 16). This enables us reason about SO(3) action using its irreducible representations encoded Wigner D-matrices (Section 3.
camera

---

(x²x)
R1
Invariance
Invariance
X2
(x2)
R2
Projected Spherical Signal
Corrected Spherical Signal
Scene
Camera RGB Image
Figure 3: Illustration of Equivariance Correction. The left side shows two identical scenes under different global transformations. Since the wrist-mounted camera captures images in its local frame, the resulting images, and thus the projected spherical signals, remain identical across both scenes. By applying the gripper orientation R as equivariance correction, we align these spherical signals to common world frame, ensuring their equivariant transformation under global scene rotations.
Image Encoder Our image encoder detailed in Figure 2a. First, encode the input image / from the observation x using standard map X(T) is mapped onto the sphere using a learnable orthographic projection (see Appendix A for details). This converts a "flat" image into spherical signal ((x): @ - IR that is easier to manipulate in (3).. We represent this spherical signal the spectral domain as truncated Fourier coefficients calculated using the spherical Fourier transform (Equation 1).
image encoder X. Next, the resulting feature
Equivariance Correction At this point, the image encoding has been projected onto the sphere and represented using spherical harmonics. However, there is problem. Since global 3D transformations of the world transform the camera and objects equally, the observed image and the projected signal (xx) would be invariant. This introduces mismatch in that ((x) stays constant while the world and actions rotate, thereby breaking global (3 (3)ee))vvarrance We accommodate this by rotating the spherical signal by amount corresponding to the SO(3) orientation of the gripper. We call this the equivariance correction factor, and illustrated Figure 3. the left of Figure we see two scenes that are the same except for SO(3) rotation of g. The eye-in-hand camera image (of the banana) is the same in both situations, even though the scene is rotated. This results in the identical projected spherical signal. It is only by applying the equivariance correction factor to the two respective signals (R₁ and R2) that we recover the camera pose the spherical signal. This ensures that the spherical signals produced in different camera poses represented in consistent global frame. We analyze this approach below.
Definition 1 (Equivariance Correction). Ler G be a group acting the input space X and output space Y. For a function : -Y, an Equivariance-Cone Correctiin map is any C: X - G satisfying C(g.x) f(g=z) g(x)) (() for all g gEG, and E X. The corrected function feor(()) C(x) f(x) is therefore G-equivariant.
Notice that Definition 1 implies (x) and fgxx are in the same orbit. Equivariance Correction is similar canonicalization map c - G, where fcano = c(x) (c(a)))) transforms the input to canonical frame, then transforms the output back to the original frame. When f(x) f(gz), Equivariance Correction a special case of canonicalization where fc((x)-1x) *x== = f(x) is invariant, so it only transforms the output to restore equivariance without altering the input.
We now show that Definition is satisfied when the correction map is chosen to be the end-effector rotation. Let x E X denote the robot observation at given timestep, which includes an eye-in-hand RGB image I and the corresponding camera (end-effector) pose R₂ SO(3) in the world frame. Let (xx) denote the spherical signal derived from the image I, expressed in the camera frame, and let p representation SO(3) acting on (xx).
Proposition 1 (Equivariance Correction via End-Effector Pose). The map C: (I, Rz) Rz, which assigns each camera image to its corresponding camera pose R2 SO(3) an equivariance correction. The corrected signal &b (eor(()) = p(C(2)))((z) p(Rz) is in a world-aligned frame. Thus, the mapping & corr is (0(3) -eeuvivrirrra for any global rotation g E SO(3), we have o &com con(gga0 x) = p(g) & &com (I).
5

---

Proof. Let g E S3(3) be global rotation applied simultaneously to the scene and the camera. Since the image is recorded in the camera frame, the spherical signal is unaffected, i.e. $(g-x) (zx). while the camera pose updates as R2 1> Rgz g For the corrected signal, we therefore obtain L (go)( = p( ) ((x)) =
p(g) (z).
(4)
where the second equality follows from the homomorphism property p(g - plg)p(RR) of the representation p. Hence the map ecor is -equiivrrir
A concrete realization of p with the spherical- harmonic coefficients and Wigner D-matrices is given in Appendix B, where the proposition reduces to the rotation of coefficient vectors in Eq. 2.
Camera-rotation invariance Our model also enforces an additional symmetry, rotations of the camera around its optical axis while the object remains stationary. These rotations form an (0( subgroup. Such rotations transform both the image and the camera pose, but their effects cancel out in the corrected world felddffamme signal. We now formalize the invariance of the corrected world-frame signal under such transformations.
Proposition 2 (Invariance to SO(2) Rotation of the Eye-in-hand Camera). Lel g E SO(2) be rotation about the camera's 's optical axis. Then, under the transformation (I, Rzz) + (g. 1, ) the corrected signal defined in Proposition I remains invariant: con (g x) 2 (x)
Proof. Assume the image encoder À is -qquivaaaaa, i.e., M(g- 1) = g X(1) for all g E SO()).
Because spherical projection and spherical Fourier transform preserve equivariance, the spherical signal satisfies ((gg z) g ((z). Meanwhile, the camera pose transforms as R.g + Ræg since its world-frame orientation Rg by g (i.e., rotating the camera relative to itself). The corrected signal is:
applying an (2( rotation 9 in the camera frame corresponds to right-multiply
Peoo (g- x) =
&gg 2) = pl Ren ') p(g) (x) = RR,, ((x) = 4 Pear(x).
Thus, the corrected signal invariant under any SO( rotations of the eye-in- hand camera.
By combining Propositions I and 2, we obtain a two-level symmetry in the encoder: the features are globally (333)-cquvvariant and locally (0n2)-invariant to rotations of the camera around its optical axis. These properties are inherently preserved without requiring additional constraints. As shown in Section 5, encoding these properties into the network leads to empirically improved performance.
4.2 SO(3) Equivariant Diffusion
As described in Section 3.3, we enforce end-to-end -qqiivarrree by requiring the denoising network I to satisfy: T(g O,g a² k) = g I(O, a ,)) for all g E SO(3). To achieve this, we extend the 2D denoising model from EquiDiff [65] to 3D. EquiDiff applies a shared ID temporal U-Net [49] independently to each group element C. C SO(2). This element-wise weight sharing guarantees that the same parameters act every group element, resulting in noise embedding in the regular representation. To generalize to 3D, we approximate the continuous symmetry group SO(3) with a finite subgroup and perform sampling accordingly. Denote H c SO(3) the subgroup that the diffusion process is equivariant to (c.g., the icosahedral group I60). Denote S C SO(3) a set that is closed under H, i.c., HS S. Intuitively, S could be viewed copies of the rotations in H, each with different offset angles to capture a denser discrete signal. Given a signal Tp SO(3) + Rd each sample (()(S) = (I (s.)) : s; E 5). where both the input and output can be treated as copies of the regular representations of H. Since g E H permutes the order of (S) and (S)) identically, the entire process is H-equivariant. Because the spherical convolution layers output signal on SO(3), we can flexibly choose any finite group H and sampling set S for discretization. In our implementation, we use both Cg C SO(2) and Ico C SO(3) as choices of H. We refer readers to Appendix C for further details.
first sample (SS) { (s:) 8, E S) and then evaluate the U-Net pointwise
4.3 End-to-End Symmetry Analysis
In this section, we analyze the equivariant properties of our method. First, due to the (0(3))equivarian encoder (Proposition 1) and the SO(3)-equivariant diffusson model (Sec- tion 4.2), our policy has end-to-end symmetry to global scene SO(3) rotations. This signifi- cantly improves its sample efficiency and generalizability to world coordinate frame changes.

---

(1) Square D2
(2) Hammer Cleanup D1
(3) Coffee Preparation D1
(4) Stack Three D1
Figure 5: A subset of experimental environments from MimicGen. Left: external view of the task.
Right: eye-in-hand observation used in the experiments. The full set of tasks is shown in Appendix D.
The benefit of (2) camera-rotation invariance (Proposi- tion 2) is subtle. Under a rotation of the gripper with respect to the workspace, no a priori constraint can be placed on how the action trajectory should transform. However, our diffu- sion model receives representation from the observation encoder that equivariant to this rotation because it con- structed from invariant features (from the spherical signals) and equivariant features (from the end-effector rotation), thus providing structured geometric bias. Figure 4 illustrates the benefit of this design. In both states (a) and (b), the gripper (triangle) aims reach the same goal pose (star), but (b) it rotated by 90 around its optical axis. Translationally, the action in (b) should remain invariant (red dots), while rotationally, should gradually transition from equivariant (yellow) invariant (green) behavior. The equivariant component in the representation ensures that the model can correctly handle the initial 90" rotation through its symmetry, while the invariant component provides stability and goal alignment. Together, this representation offers a geometric inductive bias for learning such trajectories, whereas non-equivariant models must infer these patterns purely from data. The advantage is empirically validated in Section
(a)
(b)
Figure 4: Illustration of translation
invariance and rotation equivariance-
to-invariance transition.
Experiments
5.1 Simulation
Experiment Setting We evaluate ISP twelve robotic manipulation tasks from the MinicGen benchmark [40], which is widely used in previous work on closed-loop policy learning [8, 65]. A representative subset of these simulation tasks shown in Figure (see Appendix D for a full description of all twelve MimicGen tasks). Policies are trained and evaluated exclusively using eye-in-hand RGB observations (right image in each subfigure of Figure 5). To capture sufficient scene context, we enlarge the camera's 's field of view (FOV) to approximate typical fisheye camera setup and re-generate the enlarged FOV observations using the original Mimicgen demonstrations for our method and baselines. For each task, we train three independent models with different random seeds (0, 1, and for each of the 100- and 200-demonstration settings. The models are evaluated 60 times throughout training using 50 fixed rollouts per evaluation. We report the average of the best success rates from the three runs. Task and training details are provided in Appendix D and Appendix E.
Baselines Our experiments aim to validate the benefits of explicitly modeling equivariance in eye- in-hand visuomotor policies. We evaluate two versions of ISP with different symmetry levels, 533))equivaaran version and an (2uu)eequivariant variant, which is symmetric only about rotations the plane of the table. Although the SO(3) version has more symmetry, the SO(2) version is more lightweight, which may be preferable in some settings. We compare against three strong baselines: (1) Diff'usion Policy [5]: A diffusion-based policy without any equivariance, serving as the primary reference. (2) EquiDiff (modified) [65]: Designed for fixed-camera settings, it achieves (2( equivariance via an equivariant image encoder and an equivariant temporal U-Net. For eye- in-hand control, we replace its image encoder with a standard ResNet [12], so only proprioception and denoising remain equivariant.
(3) ACT [74]: A transformer-basee behavior cloning method. To ensure fair comparison, all experiments in the following sections, including ablations and method variants, consistently apply S0(2) data augmentation during training by rotating the end-effector pose in both proprioception and actions, equivalent to jointly rotating the gripper and scene.
7

---

Table I: Success rates (%) MimicGen tasks with 100 and 200 demonstrations, averaged over seeds. We report both overall mean and per-task performance. The best result highlighted in bold, and the second best is underlined. Full results with standard deviations are in Appendix F.
Mean
Stack DI
Stuck Three DI
Square
Threading DO
Three Pc. DO
Hammer CI. D1
Method
100
200
100
200
100
200
100
200
100
200
100
200
100
200
(3P-SO(3) (+11.6 75.0 (-10.5)
100
70
88
90
92
79
73
(SP-SO(2) (+104) 73.1 (+8.6)
98
100
75
88
32
51
85
87
75
80
71
73
DifiPo
53.6
64.1
91
96
43
77
12
25
77
87
73
73
59
63
EquiDilI
53.0
64.5
96
99
61
80
9
19
89
92
74
79
59
74
ACT
23.0
40.9
45
77
12
37
3
10
36
53
28
50
35
63
Mug CI. D1
Coffee D2
Kitchen DI
Pick Place DO
Coffee Prep. DI
Nut Asse. D0
Method
100
200
100
200
100
200
(SP:SO(3)
54
64
69
75
79
42
66
41
75
82
ISP-SO(2)
52
22
46
74
84
DiffPo
55
36
EquiDiff
47
61
28
40
56
ACT
21 35
51
9
14
16
37
49
Results Table I reports the maximum success rates across all methods and configurations. In terms of performance, (3)-SO(3) achieves the best results in 21 out of 24 task settings, consistently outperforming the baselines. The remaining three settings show only marginal differences (within 1 1-2%), all within the standard error margins. Similarly, ISP-SO(2) outperforms baselines in 20 settings, which further validates the effectiveness of our design. With only 100 demonstrations, model exceeds the best- best-performing baseline by an average of 11.6%. With 200 demonstrations, the advantage remains similar at 10.5%. Importantly, our model trained with 100 demonstrations surpasses all baselines trained with 200 demonstrations and additional data augmentation, clearly demonstrating superior data efficiency. These results collectively highlight that the explicit modeling of equivariance is the key factor driving both the improved performance and enhanced sample efficiency of our method. Appendix F provides the full experimental results with standard deviations across three random seeds.
Ablation Study To assess the contribu- tion each component of our method, we conduct ablation study on four repre- sentative tasks with 100 demonstrations: Stack Three D1, Square D2, Coffee D2, and Nut assembly DO. We evaluate the fol- lowing variants of ISP-SO(3), each corre- sponding to a core module in our design: (1) Sphere: With or without the spherical projection and spherical convolutions for extracting -qquivariai features from images. (2) EquiEnc: With or without the proposed equivariant image encoder that captures (O(2)-invarian) features (Proposition 2). (3) EquiU: With or without an equivariant temporal denoising U-Net in the diffusion module. The results are sum- marized in Table 2. Removing the spherical projection leads to the largest performance drop of 9.2%, highlighting
Table 2:
Ablation study results. A red cross indicates that the corresponding module is removed in that variant.
Sphere EquiEne EquiU Sta. Cof. Nut. Squ.
Mcan
63.3 61.3 59.0 23.3 51.8 (-9.2)
66.0 57.3 61.3 32.0
54.2 (-6.8)
x
68.7 58.7
58.0 32.0
54.3 (-6.7)
70.0 64.0 75.3 34.7
61.0
critical role capturing symmetries, despite the use of data augmentation. Disabling the equivariant image encoder and the equivariant U-Net results in drops of 6.8% and 6.7%, respectively. These results demonstrate that all three components: spherical lifting, invariant encoding. and equivariant denoising, are essential for the overall effectiveness of our method. In addition, we further investigate the role of Equivariance Correction (Proposition 1) by comparing delta and absolute control strategies in Appendix G.
The Benefits of Pretraining While our method already benefits from explicit equivariance, we further explore whether incorporating pretrained image encoder can provide additional performance gains. Intuitively, pretraining can introduce stronger geometric priors and yield higher-quality visual features, especially beneficial in limitei regimes. To evaluate this effect, we conduct experiments on the MimicGen using 100 demonstrations and the same evaluation protocol described above. We compare the ISP)SO(2) with two variants: Pretraining, which initializes the image encoder with an
8

---

Table 3: Success rates (%) on MimicGen tasks with 100 demonstrations, comparing pretrained and scratch initialization of the equivariant image encoder. Results are averaged over three seeds. Values in parentheses indicate the performance difference between the two settings.
Method
Mean
Stack DI
Stack Three DI
Square D2
Threading DO
Three Pc. DO
Hammer Cl. DI
(2) (Pretraining)
72.1(+7.1)
98.0
81.3 (+6.6)
56.0 (+24.0)
91.3 (+6.6)
76.7 (+2.0)
72.7 (+2.0)
(2) (Scratch)
65.0
98.0
74.7
32.0
84.7
74.7
70.7
Mug CI. DI
Coffee D2
Kitchen DI
Pick Place DO Coffee Pre. DI Nut Assembly DD
(2) (Pretraining)
54.0 (-2.0)
66.7 (+8.0)
64.0 (-0.7)
56.3 (+10.6
63.3 (+16.6)
85.0(+11.3)
ISP)S0(2) (Scratch)
56.0
58.7
64.7
45.7
46.7
73.7
(1) Box-Pipe Disassembly
(2) U-Pipe Disassembly
(3) 3D-Pipe Disassembly
(4) Grocery Bag Retrieval
Figure 6: Real-world environments for evaluation. A GoPro camera is mounted on the robot's wrist to capture eye-in-hame observations. In each subfigure, the left image shows the initial state, while the right image shows the goal state. See Appendix H for detailed task descriptions.
ImageNet- 1k [50]-pretrained equivariant ResNet- 18, and Scratch, which trains the entire model from random initialization. Table 3 reports the maximum evaluation success rates.
Results show that ISP-SO(2) (Pretraining) surpasses ISP-SO(2) (Scratch) by 7.1%, indicating consis- tently improved final performance across most tasks. Moreover, the pretrained version with only 100 demonstrations achieves comparable performance to training from scratch with 200 demonstrations, further highlighting its data efficiency. These findings demonstrate the effectiveness of pretraining in providing richer and more stable visuomotor representations. Although the performance gains are marginal or absent in a few tasks, this suggests that naive pretraining may not always align perfectly with the downstream visuomotor learning objective. Developing pretraining strategies that are tailored to equivariant visuomotor policy representations is, therefore, a promising future direction.
5.2 Real World
Physical Setups Our real robot experiments use Universal Robot URS equipped with Robotiq-85 Gripper and custom-designed soft fingers. A GoPro camera is mounted on the wrist, following prior setups [6, 11, 35]. Demonstrations are collected via the Gello teleoperation interface [68], with observations and actions recorded 5 Hz. Following [5, 65], we employ DDIM [55] to reduce the number of denoising steps to 16. Figure illustrates the four real-world manipulation tasks. The first three tasks involve pipe disassembly, each focusing on different challenges in closed-loop control: Background-objer segmentation (Box-Pipe), long-horizon control (U-Pipe), and handling complex 3D geometries (3D-Pipe). The fourth task involves retrieving objects from deformable grocery bag, for which wrist-mounted camera observations are the only reliable source of visual information due severe occlusions and limited external visibility. We compare ISP-SO(3) against the Diffusion Policy [5]. Further details on the physical setup, task visualization, goal specification, and practical guidelines for data collection are provided in Appendix H and Appendix
Results Table reports success rates over 20 trials per task. Our method consistently outperforms the Diffusion Policy [5] baseline, with significant improvements on Box- (80% vs. 10%) and 3D-Pipe (75% vs. 15%). The former benefits from more precise visual representations that distinguish the gray pipe from the gray box background, while the latter showcases the advantage of (O((3)-qqivaari) features for reasoning over complex 3D geometries. The U-Pipe task also shows a notable gain (85% vs. 65%), demonstrating the sustained and stable performance of our equivariant method in the long-horizon task. On the Grocery Bag task, which heavily relies on eye-in-hand perception, method achieves 95% success rate. This shows its high stability and robustness. These results confirm the effectiveness of our equivariant design in addressing diverse manipulation challenges in the real world. See Appendix for a detailed failure analysis. We further evaluate the computational efficiency of ISP in real-world settings, with a comprehensive discussion provided in Appendix K. In addition, we discuss potential limitations and practical considerations of equivariance in Appendix

---

(a) Lighting Change
(b) Background Clutter (c) Partial Camera Occlusion
Figure 7: Real-world perturbation scenarios used evaluate the robustness and generalization of our method on the Box-Pipe Disassembly task.
Robustness to Real- World Pertur-
Table 4: Real-world task performance over 20 trials. The number of demonstrations used for training each task is shown in the second row.
bations To further evaluate the ro- bustness and generalization ability of our policy, we conducted addi- tional real-world experiments on the Box-Pipe Disassembly task under various domain shifts. First, we al- tered the lighting conditions by in- troducing strong white point light source near the workspace, which substantially changed the shadows and color temperature of the scene (Figure 7a). Second, perturbed the background by placing multiple household objects on the table to create clutter (Figure 7b). Finally, to test robustness against partial occlusion, we repeatedly and briefly blocked the eye-in-hand camera by rapidly waving different objects (e.g.. a toy golf club and a flower) in front of it during policy rollout (Figure 7c). Using the same initial states and 20 rollouts per condition in the previous real-worll experiments, ISP-SO(2) achieves success rates of 85% under lighting changes, 75% with background clutter, and 85% under partial camera occlusion. For reference, the performance of ISP-SO(3) without perturbations is 80%. These results demonstrate that the proposed method generalizes well to real-world disturbances and maintains strong task performance under challenging visual conditions.
Box-Pipe
U-Pipe
3D-Pipe
Grocery Bag
# Demos
65
65
65
60
ISP-SO(3) (06/22)))) (57(17/)) (1%(22)))) (5%((1/)))
DifiPo [5] 10%(2/20) 65%((3/20) 15%(3320) 75%((1//2))
6 Conclusion
In this paper, we propose Image-to-Sphere Policy (ISP), the first S0 policy learning framework for eye-in-hand visuomotor control using only monocular RGB inputs. By lifting 2D image features onto the sphere and introducing an equivariance correction mechanism to compensate for dynamic camera viewpoints, our method achieves global (0(3)-equivvriann) and local SO(2)- invariance without relying on depth sensors multi-camers setups. This design enables robust and sample-efficient policy learning in dynamic, real-world settings. Extensive experiments both simulation and real-world tasks demonstrate that ISP consistently outperforms strong baselines, achieving higher success rates with fewer demonstrations. Our work provides general and effective algorithmic solution that is both deployable and scalable for eye-in-hand hand visuomotor learning.
Limitations Our method has several limitations for future investigation. First, we only consider single wrist-mounted RGB camera. While this view provides fine-graines local information, it lacks the global scene context that an agent-view camera could offer. Effectively combining these complementary perspectives remains an important challenge. Second, our approach models rotational cquivariance but does not address translational equivariance. This limits the model's ability to generalize to object translations within the scene. Extending the equivariance correction handle camera translations a promising direction for future work. Third, the use of equivariant networks increases training time. Although inference remains efficient, reducing training overhead through more lightweight architectures would further enhance practicality. Fourth, our current method focuses on single-arm manipulation. Extending the framework to bimanual systems, where coordination between two arms is required, is a natural next step. Finally, our method does not yet leverage vision-language models. Integrating high-level semantic understanding through vision language models could further improve generalization and task understanding in more diverse environments.
10

---

Acknowledgments
We would like to thank Rachel Lim and Andrew Cole for their assistance with the real-world experiments as well as all members of the Helping Hands Lab for their valuable discussions and feedback on the manuscript. This work was supported in part by NSF grants 2107256, 2134178. 2314182, 2409351, 2442658, and NASA grant 80NSSC19K1474.
References
[1] Johann Brehmer, Joey Bose, Pim De Haan, and Taco Cohen. EDGI: Equivariant Diffusion for Planning with Embodied Agents. arXiv preprint arXiv:2303. 12410, 2023.
[2] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068// 2022.
[3] Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build E(N)-cquivariant steerable CNNs. In International Conference on Learning Representations, 2022. URL https:// openreview http://oruummddwWeeeee
[4] Ricson Cheng, Arpit Agarwal, and Katerina Fragkiadaki. Reinforcement learning of active vision for manipulating objects under occlusions. In Conference on robot learning, pages 422 431. PMLR, 2018.
[5] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668 2023.
[6] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint 2432 2024.
[7] Zihao Dong, Alan Papalia, Leonard Jung, Alenna Spiro, Philip Osteen, Christa S Robison, and Michael Everett. Learning smooth state-dependent traversability from dense point clouds. arXiv preprint arXiv:2506.064 043666666 2025.
[8] Niklas Funk, Julen Urain, Joao Carvalho, Vignesh Prasad, Georgia Chalvatzaki, and Jan Peters. Actionflow: Equivariant, accurate, and efficient policies with spatially symmetric flow matching. arXiv preprint 24Xiv:2409.04576 2024.
[9] Chongkai Gao, Zhengrong Xue, Shuying Deng, Tianhai Liang, Siqi Yang. Lin Shao, and Huazhe Xu. Riemann: Near real-time se (3)-vquiva robot manipulation without point cloud segmentation. arXiv preprint arXiv:2403. 1946..946.. 2024.
[10] Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3D Equivariant Diffusion for Target-Aware Molecule Generation and Affinity Prediction. In The Eleventh International Conference on Learning Representations, 2023.
[11] Huy Ha, Yihuai Gao, Zipeng Fu, Jie Tan, and Shuran Song. Umi on legs: Making manip- ulation policies mobile with manipulation-cente whole-body controllers. arXiv preprint arXiv:2407. 10353, 2024.
[12] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[13] Lingshen He, Yuxuan Chen, Zhengyang Shen, Yiming Dong, Yisen Wang, and Zhouchen Lin. Efficient equivariant network. In Advances in Neural Information Processing Systems, 2021.
[14] Lingshen He, Yuxuan Chen, Zhengyang Shen, Yibo Yang, and Zhouchen Lin. Neural epdos: Spatially adaptive equivariant partial differential operator based networks. In The international conference on learning representations, 2022.

---

[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[16] Owen Howell, David Klee, Ondrej Biza, Linfeng Zhao, and Robin Walters. Equivariant single view pose prediction via induced and restriction representations. Advances in Neural Information Processing Systems, 36:47251-47263, 2023.
[17] Boce Hu, Xupeng Zhu, Dian Wang, Zihao Dong. Haojie Huang, Chenghao Wang, Robin Walters, and Robert Platt. Orbitgrasp: equivvaraa grasp learnin.. arXiv preprint urXiv:2407.008 2024.
[18] Boce Hu, Heng Tian, Dian Wang, Haojie Huang, Xupeng Zhu, Robin Walters, and Robert Platt. Push-grasp policy learning using equivariant models and grasp score optimization. IEEE Robotics and Automation Letters, 10(11:111800-11100 2025. doi: 10.1109/LRA.200 3606392.
[19] Haojie Huang, Dian Wang, Robin Walters, and Robert Platt. Equivariant transporter network. arXiv preprint urXiv:2202.094000 2022.
[20] Haojie Huang, Owen Howell, Dian Wang, Xupeng Zhu, Robin Walters, and Robert Platt. Fourier transporter: Bi-equivariant robotic manipulation in 3d. arXiv preprint a7X1v:2401.12046, 2024.
[21] Haojie Huang, Dian Wang, Arsh Tangri, Robin Walters, and Robert Platt. Leveraging sym- metries in pick and place. The International Journal of Robotics Research, 4304):550-571, 2024.
[22] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, al. mo.s: vision- language-action model with open-world generalization. arXiv preprint arXiv:2504.16554, 2025.
[23] Rishabh Jangir, Nicklas Hansen, Sambaran Ghosal, Mohit Jain, and Xiaolong Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 7(2):3046-3053, 2022.
[24] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with Diffusion for Flexible Behavior Synthesis. In International Conference on Machine Learning, pages 9915. PMLR, 2022.
[25] Mingxi Jia, Dian Wang, Guanang Su, David Klee, Xupeng Zhu, Robin Walters, and Robert Platt. Seil: cnnllation-augmentee equivariant imitation learning. arXiv preprint arXiv:2211. 01.22221101199 2022.
[26] Yunfan Jiang, Ruohan Zhang, Josiah Wong. Chen Wang. Yanjie Ze. Hang Yin, Cem Gokmen, Shuran Song, Jiajun Wu, and Li Fei-Fei. Behavior robot suite: Streamlining real-world whole- body manipulation for everyday household activities. arXiv preprint arXiv:2503. 05652, 2025.
[27] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision- robotic manipulation. In Conference on robot learning, pages 651-673 PMLR, 2018.
(28) Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2 2021.
[29) Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.1 145. 2024.
[30] David M Klee, Ondrej Biza, Robert Platt, and Robin Walters. Image to sphere: Learning cquivariant features for efficient pose prediction. arXiv preprint 17Xiv22302.13926 2023.
12

---

[31] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
[32] Yikang Li, Yeqing Qiu, Yuxuan Chen, Lingshen He, and Zhouchen Lin. Affine equivariant networks based differential invariants. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5546-5556, 2024.
[33] Yikang Li, Yeqing Qiu, Yuxuan Chen, and Zhouchen Lin. Affine steerable equivariant layer for canonicalization of neural networks. In The international conference on learning representations.
2025.
[34] Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In The Eleventh International Conference on Learning Representations, 2023, URL http://pprrreeeeeeeeeeerrrrmaaaa
[35] Fanqi Lin, Yingdong Hu, Pingyue Sheng. Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. arXiv preprint 17744 2024.
[36] Kehui Liu, Chuyue Guan, Zhongjie Jia, Ziniu Wu, Xin Liu, Tianyu Wang. Shuai Liang, Pengan Chen, Pingrui Zhang, Haoming Song, al. Fastumi: A scalable and hardware-independent universal manipulation interface with dataset. arXiv e-prints, pages arXiv-2409, 2024.
[37] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt- ldt-lb: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864 2024.
[38] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101, 2017.
[39] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298. 2021.
[40] Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310. 17596 2023.
[41] Arna Kumar Mondal, Pratheeksha Nair, and Kaleem Siddiqi. Group equivariant deep rein- forcement learning. arXiv preprint SEEI::2007.034433) 2020.
[42] Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, al. Open embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6903.6 IEEE, 2024.
[43] Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint 0rX1v:2501.09747, 2025.
[44] Aaditya Prasad, Kevin Lin, Jimmy Wu, Linqi Zhou, and Jeannette Bohg. Consistency policy: Accelerated visuomotor policies via consistency distillation. arXiv preprint 2rXivv2405.07500. 2024.
[45] Yu Qi, Yuanchen Ju, Tianming Wei, Chi Chu, Lawson LS Wong, and Huazhe Xu. Two by two: Learning multi-task pairwise objects assembly for generalizable robot manipulation. CVPR 2025, 2025.
[46] Yu Qi, Haibo Zhao, Ziyu Guo, Siyuan Ma, Ziyan Chen, Yaokun Han, Renrui Zhang, Zitiantao Lin, Shiji Xin, Yijian Huang, et Bear: Benchmarking and enhancing multimodal language models for atomic embodied capabilities. arXiv preprint arXiv:2510.0877 2025.
13

---

[47] Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint urXiv:2409.00588 2024.
[48] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal conditioned imitation learning using score-based diffusion policies. In Robotics: Science and Systems, 2023.
[49] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[50] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-255 2015.
[51] Hyunwoo Ryu, Jiwoo Kim, Hyunseok An, Junwoo Chang, Joohwan Sco, Tachan Kim. Yubin Kim, Chaewon Hwang, Jongeun Choi, and Roberto Horowitz. Diffusion-edfs: Bi-equivariant denoising generative modeling on se (3) for visual robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18007-18018 2024.
[52] Mingyo Seo, H. Andy Park, Shenli Yuan, Yuke Zhu, and Luis Sentis. Legato: Cross- embodiment imitation using a grasping tool. IEEE Robotics and Automation Letters (RA-L), 2025.
[53] Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning & modes with one stone. Advances in neural information processing systems, 35:22955-22968, 2022.
[54] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference on Robotics and Automation
(ICRA), pages 6394-6400, IEEE, 2022.
[55] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[56] Sangli Teng, William Clark, Anthony Bloch, Ram Vasudevan, and Maani Ghaffari. Lie algebraic cost function design for control on lie groups. In 2022 IEEE 61 Conference on Decision and Control (CDC), pages 1867-1874. IEEE, 2022.
[57] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-qquivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.082) 2018
[58] Chenrui Tie, Yue Chen, Ruihai Wu, Boxuan Dong, Zeyi Li, Chongkai Gao, and Hao Dong. Et-seed: Efficient trajectory-level se (3) equivariant diffusion policy. arXiv preprint arXiv:2411.03990. 2024.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[60] Robin Walters, Jinxi Li, and Rose Yu. Trajectory prediction using equivariant continuous convolution. arXiv preprint arXiv:2010.1134. 2020.
[61] Dian Wang, Mingxi Jia, Xupeng Zhu, Robin Walters, and Robert Platt. On-robot learning with equivariant models. arXiv preprint arXiv:2203.0499 2022.
[62] Dian Wang, Jung Yeon Park, Neel Sortur, Lawson LS Wong, Robin Walters, and Robert Platt. The surprising effectiveness of equivariant models in domains with latent symmetry. arXiv preprint 2211:2211100223111 2022.
14

---

[63] Dian Wang, Robin Walters, and Robert Platt. -quiivraar reinforcement learning, arXiv preprint arXiv:2203.04439, 2022.
[64] Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt. Equivariant & learning spatial action spaces. In Conference on Robot Learning, pages 1713-1723, PMLR, 2022.
[65] Dian Wang, Stephen Hart, David Surovik, Tarik Kelestemur, Haojie Huang. Haibo Zhao, Mark Yeatman, Jiuguang Wang. Robin Walters, and Robert Platt. Equivariant diffusion policy. arXiv preprint TrXXiv2407.018/2 2024.
Advances in neural [66] Maurice Weiler and Gabriele Cesa. General e (2)equivariiant steerable cnns. information processing systems, 32, 2019,
[67] Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg. Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 47(8):1087-1102, 2023.
[68] Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Pieter Abbeel. Gello: A general, lost and intuitive telcoperation framework for robot manipulators. In 2024 JEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 12156-12163. IEEE, 2024.
[69] Xiaomeng Xu, Dominik Bauer, and Shuran Song. Robopanoptes: The all-seeing robot with whole-body dexterity. arXiv preprint arXiv:2501.05420, 2025.
[70] Jingyun Yang. Zi-ang Cao, Congyue Deng. Rika Antonova, Shuran Song, and Jeannette Bohg. Equibot: Sim equiqvaaar diffusion policy for generalizable and data efficient learning. arXiv preprint arXiv:2407.01 01449, 2024.
[71] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint wrXiv:2403.03954 2024.
[72] Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleop- eration. In 2018 IEEE international conference on robotics and automation (ICRA), pages 5628-5635. leee, 2018
[73] Haibo Zhao, Dian Wang, Yizhe Zhu, Xupeng Zhu, Owen Howell, Linfeng Zhao, Yaoyao Qian, Robin Walters, and Robert Platt. Hierarchical equivariant policy via frame transfer. In Forty- second International Conference on Machine Learning, 2025. URL https://openreviee. net/forumiid-aee
[74] Tony 7. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 1 13705, 2023.
[75] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the Continuity of Rotation Representations in Neural Networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5745-5753, 2019.
[76] Xupeng Zhu, Dian Wang, Ondrej Biza, Guanang Su, Robin Walters, and Robert Platt. Sample efficient grasp learning using equivariant models. arXiv preprint orX4v22202.09668, 2022.
15

---

A Orthographic Projection Details
The orthographic projection in our method follows the approach of [30], which lifts 2D feature maps onto the unit sphere in the camera frame through a signal remapping operation. Unlike traditional geometric projections that rely on explicit camera calibration or depth information, our projection is entirely learned and does not depend on predefined 3D center or physical camera parameters.
In practice, the spherical signal is sampled on HEALPix grid, which provides equal-area, hierarchical discretization of the sphere. For each point on the sphere, we apply a learnable weighted aggregation the entire 2D feature map compute its corresponding signal value. This design allows the network to flexibly determine how spatial features are mapped onto the sphere, rather than relying on fixed projection kernels. This is particularly useful for wide FOV images, where classical orthographic lifting can introduce distortions near image boundaries. By learning the mapping, the model can implicitly compensate for such distortions. However, this also means that robustness to changes in camera intrinsics (e.g., different FOVs or lens distortions) is not explicitly enforced. A promising future direction to train the projection module under diverse intrinsic settings to support the model to learn more general and transferable projection function.
B Spectral Realization of the Equivariance Correction
In this section, we provide concrete spectral realization the equivariance correction introduced in Proposition 1, using the spherical-hane harmoonic coefficients and Wigner D-matrices.
Proof. Let x be the observation with camera pose R₂ E SO(3) and let ce(x) E R denotes spherical harmonic coefficients. Under global rotation g SO(3) applied to both the scene and the camera, the camera pose transforms as R₂ + Ryr = R.. Since the signal ((x) is expressed the local (camera) frame, the spherical coefficients remain unchanged under the global transformation, so ea cg(ge) = cr(x). Applying Equation 2 with the updated camera pose, the corrected coefficients at ge are:
Co (gx) = D' (Rgg) ce(or) = (g ce(x).
Since the Wigner D-matrices D form group representation of SO(3), they satisfy the homomor- phism property: (g) = (g)g) D' (Rz)). Substituting this, we obtain:
Ct (gz) D'(g) D (R2))
Recognizing that (t.coo(z)) Dd (Rz) (e()) by Proposition we conclude:
Cr.com (gx) = D (g) (x)
(8)
Thus, the corrected coefficients Cf,cor (x) transform equivariantly under the group action g E (O(3)).
L
This result shows that equivariance correction can implemented spectrally by left-multiplying the spherical harmonic coefficients with Wigner D-matrices according to the camera orientation. This aligns the signal, originally expressed in the camera frame, to a common world frame for consistent and equivariant downstream processing across varying viewpoints.
C Implementation of Our Policy
Our model consists of an -equiqqquiaariant observation encoder followed by an (033)-equivariant diffusion module, both implemented using escnn [3] and e3nn [57].
Given an observation z E X, the SO(2) -quuiiaqui image encoder À first maps the RGB image into a regular representation, which is then mapped to trivial representation X(I) E Raxxxx where n, h, and w denote the number of channels, height, and width, respectively. These 2D features are lifted to the sphere via orthographic projection, producing a signal (xx) on S², To account for varying viewpoints, we use the gripper orientation R2 as an equivariance correction factor to align the spherical signal into a common reference frame. In our setup, the wrist-mounted camera is rigidly attached to the gripper, the gripper orientation provides fixed proxy for the camera pose. This
16

---

approximation is sufficient for aligning the mage features with the proprioceptive signals, and any minor misalignments can be further handled by the equivariant convolution layers. The corrected signal is then processed by a sequence of $ SO(3) and SO(3) - SO(3) spherical convolution layers to generate the signal v/ (ar on S0(3). The proprioceptive state encoded using the irreps Po and P1. and passed through SO(3) (3)-euvvvari linear layers to yield Fourier coefficients of the same type as (x). These are then concatenated with the image signal to form the global conditioning the number channels and d, the feature dimension. Similarly, the noisy action chunk a" is embedded into ea E Ruxda where da denotes the number of action feature channels and n the number of time steps. An inverse FFT applied to sample both 6, and en onto discrete subgroups, either the icosahedral group Teo C SO(3) or the cyclic group Cs SO(2), producing c E Rexdo,
vector co I Ruxdo where
E Rpxda xn, where = 60 or is the number of group elements. For each group element g Leo or g C2, shared (3) or SO -eeuuqqquiarri 1-D temporal U-Net processes the action sequence eg. conditioned on the observation eg and diffusion step k. This design follows the point-wise equivariant processing strategy proposed in [65], ensuring equivariance across group elements, Finally, an equivariant decoder maps the denoised representation to the noise estimate ex
D Simulation Settings
(1) Stapk D1
Stack Three
(3) Square D2
Threading DO
(5) Three Pc, Assembly Do
Hammer Cleanup D1
( Coffee D2
(8) Mug Cleanup D1
Kitchen D1
(10) Pick Place DO
(11) Coffee Preparation D1
(12) Nut Assembly DO
Figure 8: The twelve simulation tasks from the MimicGen [40] simulator. In each subfigure, the left image shows the task scene, while the right image shows the corresponding eye-in-hard in-hand view.
Figure illustrates the twelve tasks in the MimicGen simulation. each subfigure, the left image shows the full environment scene from the agent's view, while the right image is the eye-in-hand RGB observation used by the model. Following prior work [40, 65], we set the resolution of the eye-in-hand image to 84 x 84 and adopt the same maximum episode length. To enable the wrist-mounted camera to capture more contextual information, we increase FOV from 75 130 degrees, similar to that of typical wide-angle camera.
E
Training Details
For the simulation experiments, we follow the hyperparameter settings from prior work [65, 5]. In detail, we use an observation window of two history steps for ISP-SO(3) and one step for ISP-SO(2). In both cases, the denoising network outputs sequence of 16 action steps, which used for optimization during training, while only the first 8 steps are executed during evaluation. During training, input images are randomly cropped to a resolution of 76 76, while center crop is applied at evaluation time. We train all models using the AdamW [38] optimizer with Exponential Moving Average, and adopt the DDPM [15] framework with 100 denoising steps for both training and evaluation. For all baselines, we retain their original hyperparameter settings for evaluation and only adjust the number of training steps to ensure consistency across methods, All methods are trained on the same dataset and evaluated using three random seeds.
17

---

For the real-world experiments, we use the same hyperparameters in the simulation, except that we replace DDPM with DDIM [55] for both training and evaluation, and reduce the number of denoising steps to 16 at evaluation time. However, we find that using a resolution of 76 x 76 is insufficient for fine-grained manipulation in the real world, as the extremely wide FOV from the GoPro camera causes each pixel to correspond to a relatively large spatial region in the original setting. To address this, we increase the input resolution to 224 x 224. Specifically, starting from the original 720 720 RGB image captured using GoPro with the Max Lens Mod, we apply center crop size 480 x 480, followed by resizing to 224 x 224. addition, we apply standard data augmentations, including random cropping, rotation, and color jitter, to improve the robustness of both our method and the baselines.
All models are trained on single GPUs using compute clusters and workstations equipped with multiple high-performance consumer-grann GPUs.
F
Full Simulation Experiment Results with Standard Deviations
Table 5 presents the same results Table I, with standard deviations included.
Table 5: Maximum success rates (%) on MimicGen tasks with 100 and 200 demonstrations across different methods, averaged over three random seeds. The + indicates standard deviation.
Stack
Three
Square
Threading
Method
100
200
100
200
100
200
100
200
ISP-SO(3)
99.3 + 1.2
100.0 + 0.0
70.0 4: 2.0
88.0 - 2.0
34.7 + 4.2
51.3 = 2.3
90.0 + 2.0
92.0 : 0.0
ISP-SO(2)
98.0 2.0
100.0 + 0.0
74.7 7.6
88.0 + 2.0
32.0 + 0.0
50.7 5.0
84.7 + 1.2
87.3 :
DillPo
90.7 1: 4.2
96.0 2.0
43.3 1:
76.7 1: 4.2
12.0 d: 2.0
25.3 3.1
77.3 +: 10.3
86.7 1: 7.0
EquiDif
96.0 1 0.0
98.7 1.2
61.3 1 5.0
80.0 1. 2.0
8.7 1.2
19.3 1.2
88.7 5.8
92.0 1. 2.0
ACT
45.3 + 7.6
77.3 2.3
+ 2.0
36.7 = 9.9
10.0
36.0
53.3 + 6.1
Three Pc. Assembly DO
Hammer Cleanup D1
Mug Cleanup D1
Coffee D2
Method
100
200
100
200
100
200
100
200
ISP-SO(()
70.7 + 1.2
79.3 = 1.2
66.0 + 0.0
73.3 + 12
54.0 + 8.7
58.7 2.3
64.0 - 0,0
68.7 + 3.1
ISP-SO(2)
74.7 +: 1.2
80.0 2.0
70.7 1: 1.2
73.3 1: 2.3
56.0 +: 2.0
60.7 = 1.2
58.7 4: 3.1
63.3 1: 4.2
DiffPo
72.7 4
73.3 = 2.3
58.7 + 7.6
63.3 11.7
49,3 8.3
61.0 1.7
53.3 3.1
54.7 2 4.2
EquiDiff
74.0 5.3
78.7 = 1.2
59.3 + 4.2
74.0 = 2.0
$0.7 + 2.3
62.0 0.0
47.3 43 3.1
61.3 + 2.3
ACT
28.0 d: 4.0
50.0 5.3
34.7 1: 2.3
62.7 1: 5.8
24.7 3.1
37.3 5.8
20.7
1 3.1
34.7 d: 2.3
Kitchen
Pick Place DO
Coffee Preparation D1
Nut Assembly DN
Method
100
(SP)S
75.3 +: 3.1
79.3 4.2
42.0 1: 4,4
65.7 :: 5.5
40.7 + 2.3
61.3 2.3
75.3 2.5
82.0 + 7.5
(SP-SO(2)
64.7 2.3
72.0 = 2.0
45.7
61.0 5.6
46.7 4.6
56.0 0.0
73.7 7.6
84.3 1.5
DiffPo
60.7
70.7 3.1
2.1
= 1.5
37.3 1.2
52.0
51.3 11.3:8.
62.3 1.5
EquiDiff
55.3
2.9
1.
27.3
38.7
56.3
ACT
21.3 4 1.2
50.7 3.1
8.7 17 1.5
13.7 2.5
2.3
16.0 = 2.0
36.7 1.2
49.0 + 2.0
G Invariance via Delta Control vs. Equivariance via Rotation Correction
One of the core components of our method is the rotation correction step, which aligns the spherical signals to a common reference frame to preserve (ouuivaauvee throughout the policy pipeline. A natural alternative is to remove this step and instead express actions in the moving gripper frame, referred to as delta actions in [6], which can also be interpreted as a sequence of incremental transforms. This formulation leads to SIE(3)
system, both perception and action are expressed relative to the gripper's 's local frame. This raises an important question: Is rotation correction necessary if delta actions achieve similar symmetry properties through invariance?
Empirical Evidence To investigate this, we conducted additional experiments comparing absolute and delta action control on two MimicGen tasks: Square D2 and Nut Assembly DO. Specifically, we evaluated (a) variant of our method without rotation correction that uses delta control, and (b) the original Diffusion Policy with delta control. Table 6 summarizes the results with 100 demonstrations.
18

---

Table 6: Comparison of absolute and delta control MimicGen tasks with 100 demonstrations.
Values in parentheses indicate performance differences relative to ISP-SO(2) (Absolute).
Method
Square D2
Nut Assembly DO
(SP)SO(2) (Absolute)
32
74
ISP-SO(2) (Delta, No Rotation Correction)
22 (-10)
57 (-17)
DiftPo (Absolute)
12 (-20)
51 (-23)
DiffPo (Delta)
(-18)
22 (-52)
Figure 9: Real-world experimental setup. We use a URS robot equipped with a Robotiq-85 gripper and custom-designed soft fingers. GoPro camera mounted the wrist to capture visual observations. Demonstrations are collected using the Gello teleoperation interface (bottom right).
We observe that absolute action consistently outperforms delta action. Similar trends have been reported in prior work, such as EquiDiff [65] and the Diffusion Policy [5], where delta or velocity control often results in inferior performance.
Generalization Perspective Although gripper-relative control guarantees invariance under single- camera setups, does not generalize seamlessly multi-camera or hybrid sensing configurations, where additional viewpoints can break this invariance assumption. In contrast, aligning both observa- tions and actions to shared world frame establishes consistent global reference across all sensors. This property supports more flexible sensor integration and improved generalization in complex environments with multiple or moving viewpoints.
H Details of the
Experiment
Figure 9 shows our real-world experimental setup. Demonstrations are collected using the Gello telcoperation interface [68]. While the robot telcoperated in joint space, we record end-effector actions, including position, rotation, and gripper state. Visual observations and actions recorded synchronously at each timestep.
Figure 10 illustrates the initial state distributions for each task. In Box- Box-Pipe Disassembly, two pipes with different colors are connected to a junction box, where one pipe shares the same color as the box may confuse the policy. The orientations of the pipes are randomized. In U-Pipe Disassembly, four pipe fittings are arranged
U-shape and initialized with random rotations. 3D-Pipe Disassembly, two pipes are connected with independently randomized 3D orientations. In Grocery Bag Reirieval, a toy banana randomly placed inside deformable plastic bag. The robot must reach into bag, identify and retrieve the banana, and place it into transparent container with minor positional
19

---

89
(a) Box-Pipe Disassembly (b) U-Pipe Disassembly
(c) 3D-Pipe Disassembly (d) Grocery Bag Retrieval
Figure 10: Distribution of random initial states used in the real-world experiments.
variation. All subfigures Figure initializations.
show averaged visualizations across multiple randomized
We visualize one episode for each task in Figure 11. These tasks emphasize different aspects. The pipe disassembly tasks require precise, closed-loop control to smoothly extract the pipes. This makes them particularly challenging for open-loop policies. The Grocery Bag Retrieval task highlights the importance of the eye-in-hand camera, the target object is difficult to perceive and localize using only external views.
Practical Guidelines for Data Collection
Before starting data collection on the real robot, is critical to establish predefined task execution strategy to ensure motion simplicity, efficiency, and cross-operator consistency. Such strategy typically involves defining consistent action sequences, execution ranges, and task progression patterns, helping avoid ambiguous poorly structured scenarios that may lead to robotic indecision or undesirably large amount of multimodal behavior during training.
Based experience, during data collection, demonstrations should:
1. Uniformly cover as many task-relevant initial states as possible.
2. Maintain a consistent end-effector speed within and between trajectories without interruption.
3. Avoid unnecessary stops, pauses, or other irregular motion patterns.
4. Synchronize sensing and control to minimize latency-induced artifacts.
5. Regularly verify alignment between the robot and sensors prevent drift and maintain data consistency.
After data collection, all trajectories should be automatically visually inspected detect potential issues. In particular, segments exhibiting robotic hesitation or stalling, most commonly near the beginning and end of each demonstration, behavior, should be identified and removed. Consistent inspection and pruning of low-quality data can significantly improve the stability and performance of policy learning.
well episodes containing negative low-quality
These practical steps help ensure that the collected data are clean, diverse, and informative, which ultimately enhance the robustness and generalization of learned visuomotor policies.
J Real World Experiment Failure Analysis
In the Box-Pipe Disassembly task, one of the primary failure cases arises from the inability to distinguish between the gray junction box and the gray pipe. For the original Diffusion Policy, the policy consistently misidentifies the box as the pipe to be disassembled, which triggers robot's emergency stop. While our method occasionally encounters the same issue, the failure rate is significantly lower. This suggests that our method is more data-efficient and better at learning robust visual distinctions from limited demonstrations.
20

---

Table 7: Comparison of training and inference efficiency single RTX 4090 GPU.
Method
Training Speed (rel.)
Inference Time (ms)
Real-Time Capable
DiffPo
1x
68
Yes
ISP-SO(2)
2.6x slower
63
Yes
ISP-SO(3)
5.4x slower
75
Yes
In the U-Pipe Disassembly task, a common failure mode for method and the baseline occurs when pulling the red pipe inadvertently causes a connected pipe to be extracted as well. In such cases, the robot grasps both pipes simultaneously. We consider this partial success. However, the baseline additionally suffers from incorrect orientation predictions under certain initial states, leading to more frequent failures.
In the 3D-Pipe Disassembly task, our method occasionally fails identify the correct grasp orien- tation. In contrast, the baseline struggles consistently with this issue and rarely completes the task successfully. One major contributing factor is the multimodality of the task. During data collection, it is difficult to maintain consistency in demonstration strategies because pipes can be grasped in multiple orientations. Nevertheless, by incorporating 3D symmetries, our method is more robust to such variations and generalizes better across diverse configurations.
In the Grocery Bag Retrieval task, failure cases primarily result from unsuccessful grasp attempts or inaccuracies during the placement phase. The deformable nature of the bag and the partial occlusion of the banana present additional challenges, especially under limited visual feedback.
K Computational Efficiency Analysis
In this section, provide quantitative comparisons of the computational efficiency of method during both training and inference. Our results are measured on a single RTX 4090 GPU.
Training Efficiency Compared to the original Diffusion Policy [5], ISP-SO(2) is approximately 2.6x slower, and ISP-SO(3) is approximately 5.4x slower during training. This increase is expected due to the added computational complexity of the equivariant layers. Nevertheless, the training speed remains practical for scage policy learning.
Inference Efficiency Despite the higher training cost, our method maintains high efficiency during inference, making it well-suited for real-time deployment. Table 7 summarizes the average inference time of each method in the real-world settings with 16-step DDIM sampling [55]. All methods exhibit comparable inference speeds. The SO(2) variant slightly faster than the baseline, primarily due to its lighter-weight diffusion U-Net and the use of a smaller history observation window. Although the SO(3) variant marginally slower, its inference time (~75 ms) remains close to DiffPo (~~68 ms), well within real-time control requirements (e.g.. 10Hz).
L On Limitations and Practical Considerations of Equivariance
Interaction with Real-World Asymmetries Equivariance may face challenges in manipulation scenarios where asymmetries in the physical world are important. A representative example is tasks involving asymmetric robot kinematics, such as left-right differences in reachable workspace. Although equivariance allows the model generalize across rotated scenes, joint limits are not preserved under rotation, which may lead to infeasible or suboptimal actions. Another example manipulation involving heavy objects, where gravity breaks rotational symmetry in practice. An object that is easy to manipulate in one orientation may become unstable or infeasible to lift when rotated. Despite these challenges, prior work has shown that equivariant models can remain robust in the presence of symmetry-breakee shadows [62]. These asymmetries are already encoded in the input, allowing the model learn appropriate behaviors without violating the equivariant structure. While cases where equivariance leads to performance degradation are relatively uncommon, they do highlight scenarios where symmetry- breakkng mechanisms may be beneficial. A promising future direction is to augment
factors such as visual appearance, camera pose, and
21

---

equivariant architectures with non non-equivariant components or task-specific inductive biases (e.g.. gravity-aware priors or joint-limit encodings) to better capture real-world asymmetries.
Discretization of Continuous Symmetry Groups In our framework, equivariance is enforced by sampling finite subgroups of 5O(3) (e.g., T6o Cs) and applying shared U-Net across the sampled group elements. This approximation may, in principle, introduce discrepancies for rotations outside the sampled set. Empirically, however, no significant performance degradation was observed. In fact, discrete subgroups often lead to superior performance compared to continuous irreducible representations, consistent with prior findings in equivariant learning [3, 66]. While this approach reduces the theoretical degree of symmetry, it provides a more scalable and expressive modeling strategy by avoiding the computational overhead and activation function constraints associated with continuous irreps. Similar strategies have also demonstrated strong empirical effectiveness in other roboties applications [65, 20J. To further mitigate potential limitations and improve generalization, we apply random SO(2) rotations as data augmentation strategy to the end-effector pose in both proprioceptive inputs and actions during training. Additionally, subgroup sampling is not restricted to single set: multiple sets can be employed in practice to increase angular coverage when necessary. Finally, while equivariant architectures introduce structural inductive biases, they do not inherently limit the model's ability to generalize beyond the sampled rotations. With sufficient data diversity and augmentation, the network is able to interpolate smoothly across SO(3), thereby alleviating the potential impact of discretization.
M Broader Impact
This work has several potential social impacts, both positive and negative. On the positive side, our proposed method enables more data-efficient and generalizable robot policy learning in 3D environments. This can facilitate the development of more robust and capable household robots, particularly in settings where labeled demonstrations are limited. Moreover, by leveraging geometric symmetries and closed-loop visuomotor control from wrist-mounted cameras, our method could lower the barrier for deploying autonomous robots in unstructured real-world environments, thereby expanding accessibility and utility.
However, as with many data-driven learning methods, our approach inherits limitations tied to the quality and intent of the training data. Since the robot policy is learned entirely through imitation, any unsafe, biased, or suboptimal behavior demonstrated during data collection may be reflected in the final policy. Furthermore, the increased autonomy enabled by our method underscores the importance of safety monitoring and responsible deployment, especially in applications involving human interaction.
22

---

(a) Box-Pipe Disassembly
(b) U-pipe Disassembly
23

---

(c) 3D-Pipe Disassembly
(d) Grocery Bag Retrieval
Figure 11: Visualization of one episode for each task. Each subfigure illustrates a key action step in the trajectory.
24

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly state the core claims of the paper, including our key contributions and the assumptions. These claims are well-supported by both the theoretical analysis and experimental results presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
25

---

Justification: The paper includes a discussion of the limitations of our method. Specifically, we clarify the computational efficiency issues introduced by using equivariant networks, and discuss limitations related to the task settings (e.g., single-arm manipulation and single-view observations). We also outline potential directions for future research to address these limitations and extend the applicability of our approach.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All definitions, propositions, and assumptions are clearly stated, numbered, and reosseeefere
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
26

---

Answer: [Yes]
Justification: We will submit the code as supplementary material and release the code implementation for all experiments in this paper. Detailed descriptions of our method are also provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis-
sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include the complete code for data generation and all models in the supplementary material, ensuring full reproducibility of our results. A public GitHub repository will be provided with the final version of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
public/ selliicgguiee///e www.dse/ooeeeuuieege
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // //nips wttpppaawccgi/c/eeeeeeeeeeeeeeee http://gggeeeeeeeeeeeeeiiiieeeaa
for more details.
27

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: the Appendix, we provide the training details of our method, along with the code as well.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the full experimental results with standard errors computed over multiple random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
28

---

Answer: [Yes]
Justification: We provide the details of the compute resources in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : /neurps....../ cc htte/pulic/eeeeeeeeeee heeeeiie/hhhheeeeeeeeeeeeeeeeeee ctcicccciiieeeeeess
Answer: [Yes]
Justification: Our work conforms, every respect, with the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts of our work in the appendix, which consists of both potential positive and negative societal implications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and funetioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
29

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The dataset used in this work publicly available, and our method does not pose high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: We have cited all relevant prior works, datasets, and software packages used in this paper. License, copyright information, and terms of use will be provided in our GitHub repository.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The related docs will be included in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
30

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development this research does not involve LLMS as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer LLM policy (https: httpp:://neurpsee http///iiss.c/oooffeeeeeeeeeeeee cc//Conferrnces/22225 for what should or should not be described.
31


Reviews:
[{"review_id": "dG5mmADAlM", "reviewer": "Reviewer_SQeg", "summary": "This paper proposes l diffusion policy framework that incorporates SO(3)-equivariance into eye-in-hand visuomotor learning (while previous works mostly leveraged 3D point cloud), by projecting RGB inputs onto the sphere, with rotations for camera motions. Experiments have demonstrated the performance of the proposed framework.", "questions": "See weaknesses.", "limitations": "Yes. The authors provide a clear limitations section, candidly discussing the restrictions in camera, lack of translational equivariance, single-arm policy scope, and compute cost.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The motivation of leveraging equivariance of  monocule RGB image is very good, with potential impact to the field. This setting will facilitate policy learning with broader applications. The experiments are generally good and results are promising.\n\nWeakness:\n(1) Sim2Real Transfer and Robustness: There is limited discussion/experimental evidence on how robust the representation is under severe domain shift, eg, lighting changes, partial occlusions, or unexpected camera movement/perturbations. Most real-world tasks could be regarded as within well-prepared settings.\n(2) Computational Efficiency Analysis: The supplementary mentions that equivariant networks increase training time. However, the main paper lacks quantitative analysis of inference or training speed relative to baselines, and does not address potential impacts on deployment in real-time applications.\n(3) Requirement for Segmentation. Many recent equivariant robotic manipulation models exhibit strong robustness to unsegmented inputs. In contrast, this model relies on object segmentation. While segmenting target objects is not particularly difficult these days due to open-vocab segmentation models like SAM, this requirement restricts model’s applicability to object-centric tasks. With segmentation, the model is unable to manipulate non-object entities (e.g., a pile of tiny objects as a whole) or understand global scene context.\n(4) Failure Cases and Limits: Are there specific types of manipulation (e.g., heavy occlusion, transparent objects) where the method's SO(3)-equivariance leads to failure or decreased performance?\n\nLimitation:\nThe authors provide a clear limitations section, candidly discussing the restrictions in camera, lack of translational equivariance, single-arm policy scope, and compute cost. Suggestions for improvement (e.g., incorporating more failure cases study) are constructive.", "was_revised": true, "final_justification": "Please see the comment the reviewer has raised.", "rebuttal": "The authors thank the reviewer for the thoughtful review. Please see our response below:\n\n> Sim2Real Transfer and Robustness: There is limited discussion/experimental evidence on how robust the representation is under severe domain shift, eg, lighting changes, partial occlusions, or unexpected camera movement/perturbations. Most real-world tasks could be regarded as within well-prepared settings.\n\nOur paper **does not** do any sim-to-real transfer; we consider only sim2sim and real2real train/test scenarios. This is the same setting that is used in our baselines. That said, we agree with the reviewer that analyzing the domain shift between training and testing is important. Toward that end, we have performed additional real-world experiments as follows.\nSpecifically, we varied lighting conditions by introducing a strong white point light source near the table, replacing the original overhead warm ambient lighting. This significantly altered the shadows and color temperature. Additionally, we introduced background perturbations by placing various household objects on the table to create a cluttered scene, and further increased the complexity by manually moving these objects during task execution. Finally, we introduced occlusions by briefly blocking part of the in-hand camera view with a human hand during execution. For the Box-Pipe Disassembly task, our policy achieved an **85%** success rate under the lighting change, **75%** with the noisy backgrounds, and **85%** under partial camera occlusion (each evaluated with 20 rollouts). Compared with the **80%** of no perturbation reported in Section 5.2, our method demonstrates strong generalization to realistic perturbations. We will include these results in the final version of this manuscript.\n\n> Computational Efficiency Analysis: The supplementary mentions that equivariant networks increase training time. However, the main paper lacks quantitative analysis of inference or training speed relative to baselines, and does not address potential impacts on deployment in real-time applications.\n\nWe provide the following quantitative comparisons to clarify the computational efficiency of our approach.\nFor training, our SO(2) version is approximately 2 times slower than the original Diffusion Policy, while the SO(3) version is around 4 times slower, due to the added complexity of equivariant layers.\nHowever, for inference, our method remains highly efficient and suitable for real-time deployment. On a single RTX 4090 GPU:\n\n- Diffusion Policy: ~65 ms per inference\n\n- Ours (SO(2)): ~58 ms\n\n- Ours (SO(3)): ~123 ms\n\n\nThe SO(2) variant is even faster than the baseline due to our use of a lighter-weight diffusion U-net. While the SO(3) model is slower, its inference speed is still well within real-time requirements (e.g., 10 Hz control frequency), and the use of multi-step action chunks further supports efficient closed-loop execution very well.\nWe will add these results to the final version to more fully analyze the computational efficiency of both training and inference.\n\n> Requirement for Segmentation: Many recent equivariant robotic manipulation models exhibit strong robustness to unsegmented inputs. In contrast, this model relies on object segmentation. While segmenting target objects is not particularly difficult these days due to open-vocab segmentation models like SAM, this requirement restricts model’s applicability to object-centric tasks. With segmentation, the model is unable to manipulate non-object entities (e.g., a pile of tiny objects as a whole) or understand global scene context.\n\nOur model **does not** rely on any form of object segmentation, as clarified in the final paragraph of the Related Work section. We believe this is a misunderstanding regarding our model's assumptions.  A core advantage of our approach is its ability to achieve SO(3)-equivariance directly from raw in-hand RGB inputs, without relying on object-centric representations such as segmented masks or point clouds. This stands in contrast to many prior works, where SO(3)-equivariance is typically achieved through carefully processed object-level inputs or multiple fixed, calibrated depth cameras [a, b, c]. In our framework, the in-hand camera captures the scene, and the model learns to act holistically without any manual pre-processing or object isolation. Contrary to the concern raised, we believe this design significantly enhances — rather than limits — the flexibility and generality of our approach, particularly for tasks beyond neatly segmented, object-centric scenarios. We appreciate the opportunity to clarify this point, and we will make sure to highlight this distinction more clearly in the final version.\n\n> Failure Cases and Limits: Are there specific types of manipulation (e.g., heavy occlusion, transparent objects) where the method's SO(3)-equivariance leads to failure or decreased performance?\n\nThe suggested failure cases (e.g., transparent objects or partial occlusions) are not issues for our method. In fact, in our real-world experiments, the policy successfully detects randomly placed transparent boxes and completes the task of placing objects inside them, which demonstrates the robustness to these challenging conditions.\nNevertheless, it is useful to analyze potential limitations of our method, and SO(3)-equivariance may introduce challenges in certain manipulation scenarios where asymmetries in the physical world are important. One example is tasks involving left-right distinctions in world coordinates. Although equivariance allows the model to generalize across rotated scenes, the robot’s kinematic constraints (e.g., joint limits on different arms) may not remain symmetric under rotation, potentially leading to failures. Another example is manipulation involving heavy objects, where gravity breaks rotational symmetry in practice — an object that is easy to manipulate in one orientation might become unstable when rotated. Nonetheless, as noted in [d], such equivariant models can still perform well in the presence of symmetry-breaking factors. This is because the input (e.g., visual appearance, pose, shadows, etc) already encodes those asymmetries, allowing the model to learn appropriate behaviors without violating equivariant structure.\nWhile such cases are relatively uncommon, they do highlight scenarios where symmetry-breaking mechanisms may be beneficial. A promising direction is to augment SO(3)-equivariant models with non-equivariant components or inductive biases (e.g., gravity-aware priors or joint-limit encodings) that capture task-specific asymmetries. We leave this as a promising direction for future research.\n\n\n[a] Tie, C., Chen, Y., Wu, R., Dong, B., Li, Z., Gao, C., & Dong, H. (2024). Et-seed: Efficient trajectory-level se (3) equivariant diffusion policy. arXiv preprint arXiv:2411.03990.\n\n[b] Yang, J., Cao, Z. A., Deng, C., Antonova, R., Song, S., & Bohg, J. (2024). Equibot: Sim (3)-equivariant diffusion policy for generalizable and data efficient learning. arXiv preprint arXiv:2407.01479.\n\n[c] Wang, D., Hart, S., Surovik, D., Kelestemur, T., Huang, H., Zhao, H., ... & Platt, R. (2024). Equivariant diffusion policy. arXiv preprint arXiv:2407.01812.\n\n[d] Wang, D., Park, J. Y., Sortur, N., Wong, L. L., Walters, R., & Platt, R. (2022). The surprising effectiveness of equivariant models in domains with latent symmetry. arXiv preprint arXiv:2211.09231."}, {"review_id": "AXgcH6X3zy", "reviewer": "Reviewer_gU3a", "summary": "In “3D Equivariant Visuomotor Policy Learning via Spherical Projection” the authors propose a novel approach for SO(3)-equivariant policies only using monocular RGB input from the robot's wrist camera. Specifically, the authors propose to project 2D equivariant features onto a sphere. The sphere is further rotated using the gripper orientation before being used as a condition variable for an equivariant 3D diffusion policy. The authors compare their approach against various baselines and ablate components, showing an average performance gain of more than 10%.", "questions": "- What is the performance of a policy trained using no correction but relative control?\n- What are the true statistical variances? i.e. the model should be trained with at least 3 different seeds and be evaluated across the same seed.\n- Can you provide a numerical toy example that highlights the claims made in Sec 4.3.\n- I am lacking a differentiated discussion/analysis about tasks. For example, when you want to screw or grasp something, you want to do that independent of the global gripper rotation but rather in the local frame. Differently though, when pouring something you want to follow the gripper orientation in the world frame.", "limitations": "The authors addressed reasonable limitations in the text. In addition to the mentioned limitations, I would like to point out  the dependency on a constant global robot pose for demonstration generation and evaluation.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well written.\n- The proposed method is technical sound.\n- The authors show their applicability of their method both in simulated experiments and on a real world robot. \n\nWeakness:\n- My biggest concern is the necessity of the rotation correction. If I understand the paper correctly, the authors perform the correction to align the observations with the robot in the world frame.\n> This introduces a mismatch in that (x) stays constant while the world and actions rotate, thereby breaking global SO(3)-equivariance. (L. 161f.)\n\nI would argue that this step is not necessarily needed if the action space is also changed to be within the gripper rotation frame. In such changes the whole system becomes invariant to g. As this is a fundamental pillar of the paper, I would be interested in comparing against this change. If I am not mistaken, this would be similar to EquiDiff with relative control as in their paper described. The same change should also be applied to Diffusion Policy. A discussion about it could also be relevant or other potential ways to achieve equivariance.\n- While I generally agree with eye-in-hand cameras being useful for robotic tasks, I dislike the overly bold statements that it is either eye-in-hand RGBs or PointCloud. There are no reasons why not both of them could be combined [1]. Also calibrating a camera is quite easy.\n- I lack a comparison against visual servoing baselines.\n- The anonymous repository seems broken.\n-No motivation about the used tasks, authors reference related work. \n- > evaluated with 50 rollouts per task with 3 random seeds\n\n  If I understand this sentence correctly this means 3 different evaluation seeds were used. I would consider this is insufficient as essentially this will be a 150 rollouts per task with binary results for which in that case only a confidence interval can be provided.", "was_revised": true, "final_justification": "The authors clearly clarified my doubts about using relative actions in the grippe frame and provided solid arguments for future integration.", "rebuttal": "We thank the reviewer for the insightful suggestions. Please see our response below.\n\n> My biggest concern is the necessity of the rotation correction. The authors perform the correction to align the observations with the robot in the world frame. I would argue that this step is not necessarily needed if the action space is also changed to be within the gripper rotation frame. In such changes the whole system becomes invariant to g. As this is a fundamental pillar of the paper, I would be interested in comparing against this change. If I am not mistaken, this would be similar to EquiDiff with relative control. The same change should also be applied to Diffusion Policy. A discussion about it could also be relevant or other potential ways to achieve equivariance. (What is the performance of a policy trained using no correction but relative control?)\n\nWe agree that a relative action leads to an invariant system, potentially achieving similar symmetry as our model. However, we argue that aligning both observations and actions to the world frame provides **practical** and **theoretical** advantages, supported by the following evidence:\n\n**Empirical Evidence**\n\nTo directly compare the effectiveness of relative vs. absolute control, we conducted additional experiments on two tasks from MimicGen: Square D2 and Nut Assembly D0. We evaluated a variation of our method, as the reviewer suggested, which removes the rotation correction and uses relative action. In addition, we also test the original Diffusion Policy with relative control. The results (with 100 demos) are summarized below:\n\n| 100 demos              | Square D2 | Nut Assembly D0 |   |   |\n|------------------------|-----------|-----------------|---|---|\n| Ours SO(2) (abs)       | 32        | 74              |   |   |\n| Ours SO(2) (rel)       | 22 (-10)  | 57 (-17)        |   |   |\n| Diffusion Policy (abs) | 12 (-20)  | 51 (-23)        |   |   |\n| Diffusion Policy (rel) | 14 (-18)  | 22 (-52)        |   |   |\n\nThese results suggest that absolute actions outperform relative actions, despite the symmetric benefit of the relative actions. Notice that similar findings were reported in both EquiDiff and the original Diffusion Policy paper, where relative (or velocity) control often leads to a worse performance compared to absolute control.\n\n**Theoretical Perspective**\n \nWhile gripper-relative control maintains invariance under single-camera conditions, it does not generalize seamlessly to multi-camera or hybrid sensing configurations. In such settings, the external cameras will break the invariant assumption of the whole system.  However, our world-frame alignment strategy circumvents this limitation by maintaining a **consistent global reference frame** across all observations and actions, enabling broader applicability and more robust generalization in complex environments.\n\n> I dislike the overly bold statements that it is either eye-in-hand RGBs or PointCloud. There are no reasons why not both of them could be combined. Also calibrating a camera is quite easy.\n\nWe agree that monocular RGB and point cloud inputs are not mutually exclusive and can be **complementary**. A key focus of this paper is to demonstrate how SO(3)-equivariance can be achieved purely from monocular RGB inputs, without requiring depth sensing or camera calibration. Nevertheless, it does not preclude the integration of richer sensors. In fact, a key advantage of our method is that all equivariant features are explicitly transformed into the world frame. This alignment enables seamless integration of multiple viewpoints/input modalities while preserving SO(3)-equivariance across the full observation space.\n\nWe will clarify these points and revise the bold statement in the final version to better present our method as a modular and extensible approach.\n\n> lack a comparison against visual servoing baselines\n\nVisual servoing methods fundamentally differ from our approach in both capability and scope, making them an unsuitable baseline for comparison. While these methods are a key part of the vision-based control literature, they are typically limited to free-space motion guided by visual error minimization and are not designed to handle contact-rich manipulation. For example, visual servoing can move an end-effector to touch a drawer handle, but not execute the multi-step contact dynamics required to open the drawer. In contrast, diffusion policy can learn complex manipulation policies that involve repeated contact and task-specific interactions. A now-classic example of this is the Push-T task that involves making and breaking contact many times in order to move the T into the target pose. Standard visual servo methods are not directly applicable to the Push-T task without significant additional engineering. \n\nAs a consequence of the above, and because of the additional engineering that would be required, we do not directly compare against visual servo methods. We would also note that this omission is consistent with prior diffusion literature.\n\n> No motivation about the used tasks, authors reference related work.\n\nWe chose to use the MimicGen because it offers a high-quality simulation platform with well-designed tasks that span diverse manipulation challenges. It is widely adopted in recent closed-loop policy learning works [a,b,c] and enables fair and reproducible comparisons across methods.\n\nIn the real-world setting, our four tasks are designed to emphasize different aspects of closed-loop control. The first three involve pipe disassembly tasks that test precise 2D manipulation requiring accurate pulling actions, long-horizon sequential execution across multiple pipes, and precise 3D alignment involving complex geometries, respectively. The fourth task focuses on retrieving objects from a deformable grocery bag, where egocentric wrist-mounted vision is crucial. These tasks highlight the strengths of our method in handling varied scenarios.\n\n> “evaluated with 50 rollouts per task with 3 random seeds.” If I understand this sentence correctly this means 3 different evaluation seeds were used. I would consider this is insufficient as essentially this will be a 150 rollouts per task with binary results for which in that case only a confidence interval can be provided. (What are the true statistical variances? i.e. the model should be trained with at least 3 different seeds and be evaluated across the same seed.)\n\nThe reviewer's suggestion is exactly what we did in our experiments. We apologize for the confusion caused by our original wording. To clarify: for each task, we train three independent models using different random seeds (0, 1, and 2), and each model is evaluated using the same seed. The success rates reported in Table 1 are averaged across these three runs, and the full results with corresponding variance are provided in Appendix E (Table 4). We appreciate the opportunity to clarify this and will revise the description in the final version to avoid ambiguity.\n\n> provide a numerical toy example that highlights the claims made in Sec 4.3.\n\nWe provided a numerical example in our ablation study to support the core claim in this section. As shown in Table 2 of the main paper, replacing the SO(2)-equivariant image encoder (**EquiEnc**), which is used to extract SO(2)-invariant features, with a standard CNN results in a 6.8% average drop in success rate across four tasks. This result empirically supports Proposition 2 by demonstrating the benefit of using SO(2)-invariant image features.\n\n> lack a differentiated discussion/analysis about tasks. For example, when you want to screw or grasp something, you want to do that in the local frame. Differently though, when pouring something you want to follow the gripper orientation in the world frame.\n\nWe agree with the reviewer that analyzing task requirements from a local/global frame perspective could provide additional insight into the capabilities of our method. Importantly,  our approach avoids the need for explicit canonicalization or task-specific frame selection by unifying all observations in the world frame, enabling a consistent and generalizable formulation across tasks.\n\nInspired by the reviewer's comment, we categorize the 12 simulation tasks into two groups: those are more naturally aligned with local frame (Stack D1, Stack Three D1, Square D2, and Coffee D2), and those depend more on global orientation (Threading D0, Three-Piece Assembly D0, Hammer Cleanup D1, Mug Cleanup D1, Kitchen D1, Pick and Place D0, Coffee Preparation D1, and Nut Assembly D0).\n\nUsing 100 demonstrations for training, our method, despite using a unified world-frame representation, achieves significant improvements over Diffusion Policy across both categories: a 17.25% gain for tasks better suited to local frames, and an 8.88% gain for tasks that rely more on global orientation. Compared with the baseline, the benefit of our approach is stronger on the tasks more naturally aligned with local-frame reasoning. This empirical result suggests that our SO(3)-equivariant world-frame representation is expressive enough to capture diverse task constraints while maintaining a unified and consistent formulation without requiring manual frame selection.\n\n> Paper Formatting Concerns\n\nThank you for pointing out the formatting issues. We will fix the related errors in the final version.\n\n[a] Wang, Y., Zhang, Y., Huo, M., Tian, R., Zhang, X., Xie, Y., ... & Tomizuka, M. (2024). Sparse diffusion policy: A sparse, reusable, and flexible policy for robot learning. arXiv preprint arXiv:2407.01531.\n\n[b] Wang, D., Hart, S., Surovik, D., Kelestemur, T., Huang, H., Zhao, H., ... & Platt, R. (2024). Equivariant diffusion policy. arXiv preprint arXiv:2407.01812.\n\n[c] Funk, N., Urain, J., Carvalho, J., Prasad, V., Chalvatzaki, G., & Peters, J. (2024). Actionflow: Equivariant, accurate, and efficient policies with spatially symmetric flow matching. arXiv preprint arXiv:2409.04576"}, {"review_id": "9NX9k4wXeZ", "reviewer": "Reviewer_yUji", "summary": "The paper introduces a framework that projects monocular RGB images onto a sphere and corrects for camera pose to encode SO(3)-equivariance for visuomotor policy learning. It applies spherical convolutions to these world-aligned features and uses an equivariant diffusion process to predict multi-step action sequences conditioned on observations. The authors prove that their encoder maintains global SO(3)-equivariance and local SO(2)-invariance under camera rotations by leveraging spherical harmonics and Wigner D-matrices. In simulation across twelve manipulation tasks, the SO(3)-equivariant policy outperforms diffusion policy, planar-equivariant diffusion, and transformer-based baselines in terms of success rate and sample efficiency. On a UR5 robot with a wrist-mounted GoPro, the method yields higher success rates on pipe disassembly and grocery bag retrieval tasks using 60–65 demonstrations.", "questions": "- How sensitive are the results to calibrations?\n- How were the finite subgroup and the sampling set chosen in practice, and how sensitive is performance to the size and structure of them?\n- How are camera intrinsics and lens distortions handled during the orthographic projection onto the sphere, and what happens if the FOV or intrinsics change at test time?\n- Can you clarify the preprocessing steps applied to the GoPro images and whether they differ between simulation and real-world experiments?", "limitations": "The authors have provided high-level discussions of the limitations of the method.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "+++ The paper is very well-written, with a clear thesis, problem formulation, proofs, and experiments. It's a very solid effort and self-contained, regardless of how significant or impactful the idea is.\n\n+++ The paper formally defines an “equivariance correction” mechanism using the end-effector rotation, proves global SO(3)-equivariance (Proposition 1) and local SO(2)-invariance (Proposition 2) via spherical harmonics and Wigner D-matrices, and provides concrete spectral proofs in the appendix.\n\n+++ Evaluation includes twelve diverse simulation tasks and four real-robot tasks, with comparisons against diffusion policy, planar-equivariant diffusion, and transformer baselines—showing consistent, large gains in both success rate and data efficiency. The real-world robot demo looks quite convincing.\n\n+++ Ablation studies isolate the impact of spherical projection, the SO(2)-equivariant encoder, and the equivariant diffusion U-Net, demonstrating that each component contributes significantly to overall performance (drops of 6–9% when removed).\n\n--- The method enforces equivariance by sampling a finite subgroup (e.g. I₆₀ or C₈) of SO(3) and applying a shared U-Net over those elements. This discretization may fail to capture rotations outside the sampled set, potentially degrading performance under arbitrary continuous rotations. Is this an issue during experiments?\n\n--- The orthographic lifting of a wide-FOV image onto the sphere can introduce distortions—especially near the edges—that the model must learn to correct. The paper does not evaluate robustness to changes in camera intrinsics (e.g. different FOVs or lens distortion), which could affect real-world deployment.\n\n--- One potential drawback of enforcing strong inductive biases such as symmetry and equivariance is the potential loss of generalization ability. All experiments (simulation and real) use environments and objects drawn from the same benchmarks. The paper does not assess generalization to novel tasks, varied backgrounds, lighting conditions, or object textures. I believe experiments under more diverse setting and contexts will be very helpful in assessing the significance of the proposed method.", "was_revised": true, "final_justification": "I appreciate the authors for the really detailed response. The response has addressed most of my concerns, especially the ones regarding calibration and camera intrinsic. However, my concerns for the generalization issue with newly added structure to the policy still hold. I will maintain my current rating.", "rebuttal": "We thank the reviewer for the careful review. Please see our response below.\n\n> The method enforces equivariance by sampling a finite subgroup (e.g. I₆₀ or C₈) of SO(3) and applying a shared U-Net over those elements. This discretization may fail to capture rotations outside the sampled set, potentially degrading performance under arbitrary continuous rotations.\n\nWe did not observe performance degradation due to discretization in our experiments. We empirically found that using discrete subgroups like $I_{60}$​ and $C_8$​ yielded better performance than continuous irreducible representations (irreps), which is consistent with prior findings in equivariant learning [a,b]. While discrete groups reduce theoretical symmetry, they enable scalable and expressive models by avoiding the computational overhead and the choice constraint of activation functions associated with continuous irreps. These subgroups are also empirically effective in robotics applications [c,d].\n\nTo further mitigate potential limitations of discretization and improve generalization, we apply random SO(2) rotations to both observations and robot states during training to augment the data, as described in Appendix D (Training Details).  Moreover, although each sampled set contains a finite number of rotations (e.g., 60 in $I_{60}$), we are not limited to a single set — in practice, we can sample multiple such sets to improve angular coverage when necessary.\n\nFinally, while equivariant modeling imposes useful inductive structures, it does not prevent the network from generalizing beyond the sampled rotations. The model retains the capacity to interpolate smoothly across SO(3), especially when supported by diverse data and augmentations.\n\nWe will clarify these points in the final version. Thank you for highlighting this.\n\n> The orthographic lifting of a wide-FOV image onto the sphere can introduce distortions—especially near the edges—that the model must learn to correct. The paper does not evaluate robustness to changes in camera intrinsics (e.g. different FOVs or lens distortion), which could affect real-world deployment.\n\nThis is a very insightful question. Unlike traditional methods relying on geometric modeling, our approach uses a **learnable orthographic lifting mechanism** that operates directly on feature maps. After extracting 2D features via an equivariant CNN backbone, we lift them to the sphere using a learnable weighted aggregation kernel. The lifting operation is a purely signal-level transformation that does not depend on camera intrinsics, depth, or calibration, making it naturally agnostic to FOV and lens distortion. As a result, the model can learn to compensate for distortions such as wide-FOV warping or lens variation during training.\n\nWe also provide empirical support to address this concern. In our experiments, we use different cameras for simulation and real-world setups, each with different focal lengths and fields of view. Despite this discrepancy, our method performs well in both domains, suggesting that the model successfully learns to compensate for intrinsic variations. In real-world deployment, we tested our method with several different FOV settings and observed reliable performance across these configurations. While we ultimately chose the best-performing FOV setting for evaluation, our experience suggests that the learned projection remains effective under all optical conditions. In future works, we plan to explore training with mixed camera intrinsics to improve generalization to arbitrary test-time camera configurations.\n\nWe believe this learnable and sensor-agnostic design is a strength of our method, enabling robust deployment across varied hardware setups. While we initially omitted a detailed explanation due to prior coverage in the I2S paper [e], we agree that adding clarification in the final version would be beneficial.\n\n> One potential drawback of enforcing strong inductive biases such as symmetry and equivariance is the potential loss of generalization ability. All experiments (simulation and real) use environments and objects drawn from the same benchmarks. The paper does not assess generalization to novel tasks, varied backgrounds, lighting conditions, or object textures. I believe experiments under more diverse setting and contexts will be very helpful in assessing the significance of the proposed method.\n\nTo answer this question, we conducted additional real-world experiments to evaluate the robustness of our method under **domain shifts**. Specifically, we varied lighting conditions by introducing a strong white point light source near the table, replacing the original overhead warm ambient lighting. This significantly altered the shadows and color temperature. Additionally, we introduced background perturbations by placing various household objects on the table to create a cluttered scene, and further increased the complexity by manually moving these objects during task execution. Finally, we introduced occlusions by briefly blocking part of the in-hand camera view with a human hand during execution.  For the Box-Pipe Disassembly task, our policy achieved **85%** success under lighting changes, **75%** with noisy backgrounds, and **85%** under partial camera occlusion (each evaluated with 20 rollouts). Compared with the **80%** of no perturbation reported in Section 5.2, our method demonstrates strong generalization to realistic perturbations. We will include these results in the final version.\n\nMoreover, we would like to clarify that the primary form of generalization targeted by our method is **spatial generalization**—specifically, generalizing across object and scene orientations via SO(3)-equivariance. Although task/instance-level generalization (e.g., novel tasks or objects) is not the focus of this work, our approach is fully compatible with such extensions. For instance, integrating pretrained encoders such as CLIP or DINO, or incorporating language instructions, could provide strong generalization across domains. This is a promising direction we plan to explore.\n\n> How sensitive are the results to calibrations?\n\nOur method does not require any calibration, as it only requires an eye-in-hand image input. In both simulation and real-world experiments, no camera calibration is performed. This independence from camera intrinsics and distortion parameters is a key strength of our approach, contributing to its robustness and ease of deployment across different camera setups.\n\n> How were the finite subgroup and the sampling set chosen in practice, and how sensitive is performance to the size and structure of them?\n\nWe chose the finite subgroups based on a trade-off between rotational coverage and computational efficiency. Specifically, we use the icosahedral group $I_{60}$ for SO(3), which provides one of the densest and most uniformly distributed finite approximations of the rotation group, and $C_8$ for SO(2), which is a lightweight cyclic group commonly used in prior works. For each subgroup, we use all group elements as the sampling set.\n\nIn practice, we find that performance is not highly sensitive to the exact choice of subgroup, as long as the sampling provides reasonable angular coverage. Additionally, we apply random rotation augmentations during training, which expose the model to diverse data and enhance both generalization and continuity.\n\n> How are camera intrinsics and lens distortions handled during the orthographic projection onto the sphere, and what happens if the FOV or intrinsics change at test time?\n\nOur method does not require or process camera intrinsics or lens distortion during the orthographic projection. The lifting step is a learnable mapping from the feature map onto a unit sphere, and does not rely on camera calibration, depth, or distortion parameters. This design allows the method to be sensor-agnostic and simplifies deployment.\n\nRegarding FOV or intrinsics changes at test time: similar to other behavior cloning-based visuomotor methods, our model is trained and evaluated under the same camera setup. Significant changes in FOV at test time would introduce a domain shift in the input distribution, which would affect any policy trained under fixed-camera data. However, future works could explore training with mixed camera FOV or intrinsic, enabling the model to generalize to arbitrary test time camera settings.\n\n> Can you clarify the preprocessing steps applied to the GoPro images and whether they differ between simulation and real-world experiments?\n\nThe preprocessing steps are detailed in Appendix D (see “Training Details”). For real-world experiments, we capture 720×720 RGB images using a GoPro with the Max Lens Mod, then apply a center crop to 480×480 and resize to 224×224 to reduce spatial resolution under the wide field of view. In simulation, we use a lower resolution of 76×76, which is sufficient given the simpler visual complexity and is consistent with baseline settings. Data augmentations, including cropping and rotation, are applied in both domains, while color jitter augmentation is used only in the real-world setting to improve robustness against lighting variation.\n\n[a] Cesa, G., Lang, L., & Weiler, M. (2022). A program to build E (N)-equivariant steerable CNNs. In International conference on learning representations.\n\n[b] Weiler, M., & Cesa, G. (2019). General e (2)-equivariant steerable cnns. Advances in neural information processing systems, 32.\n\n[c] Wang, D., Hart, S., Surovik, D., Kelestemur, T., Huang, H., Zhao, H., ...& Platt, R. (2024). Equivariant diffusion policy. arXiv preprint arXiv:2407.01812.\n\n[d] Huang, H., Howell, O., Wang, D., Zhu, X., Walters, R., & Platt, R. (2024). Fourier transporter: Bi-equivariant robotic manipulation in 3d. arXiv preprint arXiv:2401.12046.\n\n[e] Klee, D. M., Biza, O., Platt, R., & Walters, R. (2023). Image to sphere: Learning equivariant features for efficient pose prediction. arXiv preprint arXiv:2302.13926."}, {"review_id": "NhXB6hYeS0", "reviewer": "Reviewer_GXof", "summary": "This paper proposes a visuomotor policy learning framework that is SO(3)-equivariant for robot manipulation using an eye-in-hand monocular camera. \n\nThis approach first extracts SO(2)-equivariant features from an in-hand camera using an equivariant encoder. These features are then projected onto a sphere using spherical harmonics, enabling SO(3)-equivariant policy learning via monocular images. Next, equivariant correction is done by rotating invariant wrist-mounted image representation with gripper orientation. Finally, the authors extend EquiDiff from SO(2) to SO(3) to achieve SO(3)-equivariant visuomotor policy learning. \n\nThe main contribution is the equivariant correction step that bridges the gap between 2D equivariant policy learning using top-down image and 3D equivariant policy learning based on point cloud data.", "questions": "- What are the trade-offs between using SO(3)-equivariant features from image and using equivariant features from point cloud inputs? \n- The real-world experiments are conducted using a similar robot platform as in simulation. A discussion of sim-to-real transfer, hardware generalizability, and data collection requirements would improve the paper’s practical relevance. For instance, how does the system perform with different camera configurations, image resolutions, or frame rates? Is retraining required when switching hardware?\n- For spherical projection, what defines the center of the sphere, and how is the projection spatially interpreted?\n- The paper mentioned that the SO(3) version is more computation intensive than SO(2). Could using a sparser symmetry group improve efficiency?", "limitations": "The authors have listed out some critical limitations of this work. Additionally, monocular images inherently suffer from scale ambiguity. The paper does not address how this issue is mitigated. A discussion of this limitation would be beneficial for readers to be aware.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- The paper is clearly written and well motivated, with extensive literature review that describes current robot manipulation policy learning.\nSuperior performance improvement is shown in the MimicGen benchmark with only 100 demos and in real-world data with around 60 demos.\n- The proposed spherical projection and equivariant correction presents a practical method for extending 2D image-based manipulation policy learning to 3D equivariant settings.\n- The paper includes detailed background on necessary backgrounds such as spherical harmonics and diffusion policy. \n\nWeakness\n- The introduction and literature review of equivariant robotics manipulation describes the current research gap of using the point cloud input whereas the proposed method uses monocular image. There could be some discussion on how the proposed method compares or complements the point cloud-based approach. Additionally, Figure 2 is potentially misleading as it shows point clouds as an example even though the paper focuses on image. \n- The projection from image features to the sphere is described only as “orthographic projection”. Compared to the detailed background on representations of SO(3), the projection step lacks detail for the reader to fully understand. For example, what defines the center of the sphere, and how is the projection spatially interpreted?\n- The description in section 4.3 is confusing. Earlier sections focus on construction of SO(3) equivariant features, but this section starts with SO(2)-equivariant encoder and SO(2)-invariant features without clearly explaining how these are integrated. The benefit of combining invariant and equivariance (as shown in Figure 3) is also not convincing, especially since the network does not separate translation (which needs invariant) and rotation (needs equivariant to invariant). In addition, equivariance is the general form of invariance. The role of invariant features could be better justified.\n- The paper mentioned that the SO(3) version is more computation intensive than SO(2). Adding runtime or memory usage could be helpful to quantify the trade-off between the two versions. \n- The literature review overlooks a recent SE(3)-equivariant point cloud-based method: Zhu, Xupeng, Fan Wang, Robin Walters, and Jane Shi. \"SE (3)-Equivariant Diffusion Policy in Spherical Fourier Space.\" In Forty-second International Conference on Machine Learning.", "was_revised": true, "final_justification": "All of my questions have been addressed. I appreciate the clarification on the comparison between learning SO(3)-equivariant features from images versus equivariant features from point clouds. With all the corrections and additional information, I maintain my positive rating for this paper.", "rebuttal": "We thank the reviewer for the helpful comments. Please see our response below.\n\n> The introduction and literature review of equivariant robotics manipulation describes the current research gap of using the point cloud input whereas the proposed method uses monocular image. There could be some discussion on how the proposed method compares or complements the point cloud-based approach. Additionally, Figure 2 is potentially misleading as it shows point clouds as an example even though the paper focuses on image.\n\nOur method is designed to address the unique challenge of learning SO(3)-equivariant policies directly from monocular RGB inputs, in contrast to prior SO(3)-equivariant approaches that rely on point clouds [a,b]. Point cloud methods typically require accurate depth sensing, 3D calibration, and are computationally expensive due to heavy geometric processing. In contrast, our method provides a lightweight, calibration-free alternative that achieves strong performance using only monocular images. We view these approaches as complementary: point clouds offer richer geometry, while image-based methods are more scalable and easier to deploy. We will revise the related work section to clarify this distinction. Additionally, we appreciate the feedback on Figure 2 and will update it to avoid confusion.\n\n> The projection from image features to the sphere is described only as “orthographic projection”. Compared to the detailed background on representations of SO(3), the projection step lacks detail for the reader to fully understand. For example, what defines the center of the sphere, and how is the projection spatially interpreted? ( For spherical projection, what defines the center of the sphere, and how is the projection spatially interpreted?)\n\nThe orthographic projection in our paper refers to a signal remapping operation that lifts 2D feature maps onto the unit sphere in the camera frame. Unlike traditional geometric projections that require depth or camera calibration, our approach does not rely on a 3D center or physical camera parameters. The projection is learned. In practice, the spherical signal is sampled using a HEALPix grid, which provides an equal-area, hierarchical discretization of the sphere. For each point on the sphere, we apply a learnable weighted aggregation over the full 2D feature map to compute the signal value. This allows the model to flexibly adapt how spatial features are projected onto the sphere, rather than relying on fixed kernels.\n\nWe acknowledge that our initial explanation was insufficient, and we will expand this section in the final version to describe the projection mechanism. Thank you for pointing this out.\n\n> The description in section 4.3 is confusing. Earlier sections focus on construction of SO(3) equivariant features, but this section starts with SO(2)-equivariant encoder and SO(2)-invariant features without clearly explaining how these are integrated. The benefit of combining invariant and equivariance (as shown in Figure 3) is also not convincing, especially since the network does not separate translation (which needs invariant) and rotation (needs equivariant to invariant). In addition, equivariance is the general form of invariance. The role of invariant features could be better justified.\n\nBefore addressing your question, we would like to clarify a typo in Section 4.3 (line 240): “Combining the SO(2)-equivariant encoder (Proposition 1).” This should instead be an SO(3)-equivariant encoder, as our method is designed to achieve end-to-end SO(3) equivariance. We acknowledge that this error may have caused confusion regarding the motivation and design described in this section, and we will correct it in the final version.\n\nOur method maintains end-to-end SO(3)-equivariance by combining an SO(3)-equivariant encoder with an SO(3)-equivariant diffusion U-Net. This ensures that predicted actions transform consistently under global scene rotations, enabling strong generalization and sample efficiency.\n\nIn parallel, we use an SO(2)-equivariant image encoder to extract SO(2)-invariant features prior to spherical projection. These invariant features are not responsible for preserving the overall SO(3)-equivariance of the policy. Rather, we adopt this design because the resulting image encoder performs better at extracting robust and consistent visual features. In practice, this leads to improved image representations without compromising the model’s equivariant structure.\n\nThis design choice is empirically supported: as shown in the ablation study, replacing the SO(2)-equivariant image encoder (**EquiEnc**) with a standard CNN leads to a 6.8% average drop in success rate across four tasks. This confirms the practical benefit of using invariant features for image encoding. We will revise Section 4.3 to clarify this point and better explain the motivation behind using SO(2)-invariant image features.\n\n> The paper mentioned that the SO(3) version is more computation intensive than SO(2). Adding runtime or memory usage could be helpful to quantify the trade-off between the two versions.(The paper mentioned that the SO(3) version is more computation intensive than SO(2). Could using a sparser symmetry group improve efficiency?)\n\nThe SO(3) version indeed incurs higher computational cost compared to the SO(2) variant. Specifically, the SO(2) model trains about 2x slower than the original Diffusion Policy, while the SO(3) model is roughly 2x slower and requires 1.8x more memory than the SO(2) version.\n\nHowever, for inference, our method remains highly efficient and suitable for real-time deployment. On a single RTX 4090 GPU:\n\n- Diffusion Policy: ~65 ms per inference\n- Ours (SO(2)): ~58 ms\n- Ours (SO(3)): ~123 ms\n\nThe SO(2) variant is even faster than the baseline due to our use of a lighter-weight diffusion U-net. While the SO(3) model is slower, its inference speed is still well within real-time requirements (e.g., 10 Hz control frequency), and the use of multi-step action chunks further supports efficient closed-loop execution very well.\n\nFor the question of whether using a sparser symmetry group could improve efficiency: yes, our framework is fully compatible with such groups. We use the icosahedral group ($I_{60}$) as the default due to its strong trade-off between rotational coverage and performance. However, sparser groups like the octahedral can be employed to further reduce computational overhead.\n\nWe will include detailed comparisons of training time, inference time, and memory usage in the final version.\n\n> The literature review overlooks a recent SE(3)-equivariant point cloud-based method: Zhu, Xupeng, Fan Wang, Robin Walters, and Jane Shi. \"SE (3)-Equivariant Diffusion Policy in Spherical Fourier Space.\" In Forty-second International Conference on Machine Learning.\n\nThank you for pointing out this important related work. We acknowledge that Zhu et al. (ICML 2024) also study SE(3)-equivariant policy learning via diffusion in spherical Fourier space, but using point cloud input. We will add a discussion of this method in the final version and clarify how our approach differs.\n\n> What are the trade-offs between using SO(3)-equivariant features from image and using equivariant features from point cloud inputs?\n\nCompared to point cloud-based SO(3)-equivariant methods, our image-based SO(3)-equivariant framework requires no depth sensing, camera calibration, or 3D reconstruction. This makes it easier to deploy in real-world scenarios, particularly for eye-in-hand setups with limited sensing.\n\nWhile point clouds provide richer 3D geometry and enable more precise SO(3)-equivariance, they typically rely on accurate depth sensors and careful calibration. Nonetheless, we believe that leveraging monocular RGB and point cloud inputs to achieve SO(3)-equivariance is not mutually exclusive but complementary.\n\n> The real-world experiments are conducted using a similar robot platform as in simulation. A discussion of sim-to-real transfer, hardware generalizability, and data collection requirements would improve the paper’s practical relevance. For instance, how does the system perform with different camera configurations, image resolutions, or frame rates? Is retraining required when switching hardware?\n\nWe agree that discussing hardware generalizability and deployment conditions helps improve the practical relevance of our method. However, our method does not perform sim-to-real transfer; we focus on the same domain training and evaluation.\n\nA key advantage of our approach is that it does not rely on any camera intrinsics, depth sensing, or calibration. The spherical projection is a model-agnostic transformation applied to image features and operates independently of specific sensor parameters. This design makes our method naturally adaptable to different camera configurations.\n\nFurthermore, our method can generalize across different robot arms without retraining, as long as the same gripper is used. Since part of the gripper naturally appears in the in-hand camera view, maintaining a consistent gripper appearance should suffice to preserve the validity of the learned policy. If the gripper changes, a small number of additional demonstrations may be required to adapt the model.\n\nRegarding data collection requirements, we will include additional discussion based on our extensive experimental experience. This will cover practical guidelines for efficiently collecting demonstration data, strategies for increasing diversity and informativeness of samples, and preprocessing steps for trajectory handling.\n\n[a] Tie, C., Chen, Y., Wu, R., Dong, B., Li, Z., Gao, C., & Dong, H. (2024). Et-seed: Efficient trajectory-level se (3) equivariant diffusion policy. arXiv preprint arXiv:2411.03990.\n\n[b] Yang, J., Cao, Z. A., Deng, C., Antonova, R., Song, S., & Bohg, J. (2024). Equibot: Sim (3)-equivariant diffusion policy for generalizable and data efficient learning. arXiv preprint arXiv:2407.01479."}]

Author rebuttal:

Meta-review: The idea of leveraging SO(3) equivariance to monocular RGB images is particularly relevant to robotics settings, where cameras are attached to moving arms. The authors develop a method based on spherical projection and an equivariant diffusion process to predict multistep action sequences conditioned on observations. The method and experiments are comprehensive and described clearly, and all reviewers have provided substantive technical reviews and engaged with the authors in resolving any issues and concerns. I believe the review scores accurately indicate the technical impact and significance of the contributions.
Decision: Accept (spotlight)