Title: 3D Equivariant Visuomotor Policy Learning via Spherical Projection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Equivariance

Visuomotor Policy Learning

Robotic Manipulation
Abstract: Equivariant models have recently been shown to improve the data efficiency of diffusion policy by a significant margin. However, prior work that explored this direction focused primarily on point cloud inputs generated by multiple cameras fixed in the workspace. This type of point cloud input is not compatible with the now-common setting where the primary input modality is an eye-in-hand RGB camera like a GoPro. This paper closes this gap by incorporating into the diffusion policy model a process that projects features from the 2D RGB camera image onto a sphere. This enables us to reason about symmetries in $\mathrm{SO}(3)$ without explicitly reconstructing a point cloud. We perform extensive experiments in both simulation and the real world that demonstrate that our method consistently outperforms strong baselines in terms of both performance and sample efficiency. Our work, $\textbf{Image-to-Sphere Policy}$ ($\textbf{ISP}$), is the first $\mathrm{SO}(3)$-equivariant policy learning framework for robotic manipulation that works using only monocular RGB inputs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a visuomotor policy learning framework that is SO(3)-equivariant for robot manipulation using an eye-in-hand monocular camera. \n\nThis approach first extracts SO(2)-equivariant features from an in-hand camera using an equivariant encoder. These features are then projected onto a sphere using spherical harmonics, enabling SO(3)-equivariant policy learning via monocular images. Next, equivariant correction is done by rotating invariant wrist-mounted image representation with gripper orientation. Finally, the authors extend EquiDiff from SO(2) to SO(3) to achieve SO(3)-equivariant visuomotor policy learning. \n\nThe main contribution is the equivariant correction step that bridges the gap between 2D equivariant policy learning using top-down image and 3D equivariant policy learning based on point cloud data.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- The paper is clearly written and well motivated, with extensive literature review that describes current robot manipulation policy learning.\nSuperior performance improvement is shown in the MimicGen benchmark with only 100 demos and in real-world data with around 60 demos.\n- The proposed spherical projection and equivariant correction presents a practical method for extending 2D image-based manipulation policy learning to 3D equivariant settings.\n- The paper includes detailed background on necessary backgrounds such as spherical harmonics and diffusion policy. \n\nWeakness\n- The introduction and literature review of equivariant robotics manipulation describes the current research gap of using the point cloud input whereas the proposed method uses monocular image. There could be some discussion on how the proposed method compares or complements the point cloud-based approach. Additionally, Figure 2 is potentially misleading as it shows point clouds as an example even though the paper focuses on image. \n- The projection from image features to the sphere is described only as “orthographic projection”. Compared to the detailed background on representations of SO(3), the projection step lacks detail for the reader to fully understand. For example, what defines the center of the sphere, and how is the projection spatially interpreted?\n- The description in section 4.3 is confusing. Earlier sections focus on construction of SO(3) equivariant features, but this section starts with SO(2)-equivariant encoder and SO(2)-invariant features without clearly explaining how these are integrated. The benefit of combining invariant and equivariance (as shown in Figure 3) is also not convincing, especially since the network does not separate translation (which needs invariant) and rotation (needs equivariant to invariant). In addition, equivariance is the general form of invariance. The role of invariant features could be better justified.\n- The paper mentioned that the SO(3) version is more computation intensive than SO(2). Adding runtime or memory usage could be helpful to quantify the trade-off between the two versions. \n- The literature review overlooks a recent SE(3)-equivariant point cloud-based method: Zhu, Xupeng, Fan Wang, Robin Walters, and Jane Shi. \"SE (3)-Equivariant Diffusion Policy in Spherical Fourier Space.\" In Forty-second International Conference on Machine Learning.\n\nQUESTIONS:\n- What are the trade-offs between using SO(3)-equivariant features from image and using equivariant features from point cloud inputs? \n- The real-world experiments are conducted using a similar robot platform as in simulation. A discussion of sim-to-real transfer, hardware generalizability, and data collection requirements would improve the paper’s practical relevance. For instance, how does the system perform with different camera configurations, image resolutions, or frame rates? Is retraining required when switching hardware?\n- For spherical projection, what defines the center of the sphere, and how is the projection spatially interpreted?\n- The paper mentioned that the SO(3) version is more computation intensive than SO(2). Could using a sparser symmetry group improve efficiency?\n\nLIMITATIONS:\nThe authors have listed out some critical limitations of this work. Additionally, monocular images inherently suffer from scale ambiguity. The paper does not address how this issue is mitigated. A discussion of this limitation would be beneficial for readers to be aware.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework that projects monocular RGB images onto a sphere and corrects for camera pose to encode SO(3)-equivariance for visuomotor policy learning. It applies spherical convolutions to these world-aligned features and uses an equivariant diffusion process to predict multi-step action sequences conditioned on observations. The authors prove that their encoder maintains global SO(3)-equivariance and local SO(2)-invariance under camera rotations by leveraging spherical harmonics and Wigner D-matrices. In simulation across twelve manipulation tasks, the SO(3)-equivariant policy outperforms diffusion policy, planar-equivariant diffusion, and transformer-based baselines in terms of success rate and sample efficiency. On a UR5 robot with a wrist-mounted GoPro, the method yields higher success rates on pipe disassembly and grocery bag retrieval tasks using 60–65 demonstrations.\n\nSTRENGTHS AND WEAKNESSES:\n+++ The paper is very well-written, with a clear thesis, problem formulation, proofs, and experiments. It's a very solid effort and self-contained, regardless of how significant or impactful the idea is.\n\n+++ The paper formally defines an “equivariance correction” mechanism using the end-effector rotation, proves global SO(3)-equivariance (Proposition 1) and local SO(2)-invariance (Proposition 2) via spherical harmonics and Wigner D-matrices, and provides concrete spectral proofs in the appendix.\n\n+++ Evaluation includes twelve diverse simulation tasks and four real-robot tasks, with comparisons against diffusion policy, planar-equivariant diffusion, and transformer baselines—showing consistent, large gains in both success rate and data efficiency. The real-world robot demo looks quite convincing.\n\n+++ Ablation studies isolate the impact of spherical projection, the SO(2)-equivariant encoder, and the equivariant diffusion U-Net, demonstrating that each component contributes significantly to overall performance (drops of 6–9% when removed).\n\n--- The method enforces equivariance by sampling a finite subgroup (e.g. I₆₀ or C₈) of SO(3) and applying a shared U-Net over those elements. This discretization may fail to capture rotations outside the sampled set, potentially degrading performance under arbitrary continuous rotations. Is this an issue during experiments?\n\n--- The orthographic lifting of a wide-FOV image onto the sphere can introduce distortions—especially near the edges—that the model must learn to correct. The paper does not evaluate robustness to changes in camera intrinsics (e.g. different FOVs or lens distortion), which could affect real-world deployment.\n\n--- One potential drawback of enforcing strong inductive biases such as symmetry and equivariance is the potential loss of generalization ability. All experiments (simulation and real) use environments and objects drawn from the same benchmarks. The paper does not assess generalization to novel tasks, varied backgrounds, lighting conditions, or object textures. I believe experiments under more diverse setting and contexts will be very helpful in assessing the significance of the proposed method.\n\nQUESTIONS:\n- How sensitive are the results to calibrations?\n- How were the finite subgroup and the sampling set chosen in practice, and how sensitive is performance to the size and structure of them?\n- How are camera intrinsics and lens distortions handled during the orthographic projection onto the sphere, and what happens if the FOV or intrinsics change at test time?\n- Can you clarify the preprocessing steps applied to the GoPro images and whether they differ between simulation and real-world experiments?\n\nLIMITATIONS:\nThe authors have provided high-level discussions of the limitations of the method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn “3D Equivariant Visuomotor Policy Learning via Spherical Projection” the authors propose a novel approach for SO(3)-equivariant policies only using monocular RGB input from the robot's wrist camera. Specifically, the authors propose to project 2D equivariant features onto a sphere. The sphere is further rotated using the gripper orientation before being used as a condition variable for an equivariant 3D diffusion policy. The authors compare their approach against various baselines and ablate components, showing an average performance gain of more than 10%.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well written.\n- The proposed method is technical sound.\n- The authors show their applicability of their method both in simulated experiments and on a real world robot. \n\nWeakness:\n- My biggest concern is the necessity of the rotation correction. If I understand the paper correctly, the authors perform the correction to align the observations with the robot in the world frame.\n> This introduces a mismatch in that (x) stays constant while the world and actions rotate, thereby breaking global SO(3)-equivariance. (L. 161f.)\n\nI would argue that this step is not necessarily needed if the action space is also changed to be within the gripper rotation frame. In such changes the whole system becomes invariant to g. As this is a fundamental pillar of the paper, I would be interested in comparing against this change. If I am not mistaken, this would be similar to EquiDiff with relative control as in their paper described. The same change should also be applied to Diffusion Policy. A discussion about it could also be relevant or other potential ways to achieve equivariance.\n- While I generally agree with eye-in-hand cameras being useful for robotic tasks, I dislike the overly bold statements that it is either eye-in-hand RGBs or PointCloud. There are no reasons why not both of them could be combined [1]. Also calibrating a camera is quite easy.\n- I lack a comparison against visual servoing baselines.\n- The anonymous repository seems broken.\n-No motivation about the used tasks, authors reference related work. \n- > evaluated with 50 rollouts per task with 3 random seeds\n\n  If I understand this sentence correctly this means 3 different evaluation seeds were used. I would consider this is insufficient as essentially this will be a 150 rollouts per task with binary results for which in that case only a confidence interval can be provided.\n\nQUESTIONS:\n- What is the performance of a policy trained using no correction but relative control?\n- What are the true statistical variances? i.e. the model should be trained with at least 3 different seeds and be evaluated across the same seed.\n- Can you provide a numerical toy example that highlights the claims made in Sec 4.3.\n- I am lacking a differentiated discussion/analysis about tasks. For example, when you want to screw or grasp something, you want to do that independent of the global gripper rotation but rather in the local frame. Differently though, when pouring something you want to follow the gripper orientation in the world frame.\n\nLIMITATIONS:\nThe authors addressed reasonable limitations in the text. In addition to the mentioned limitations, I would like to point out  the dependency on a constant global robot pose for demonstration generation and evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes l diffusion policy framework that incorporates SO(3)-equivariance into eye-in-hand visuomotor learning (while previous works mostly leveraged 3D point cloud), by projecting RGB inputs onto the sphere, with rotations for camera motions. Experiments have demonstrated the performance of the proposed framework.\n\nSTRENGTHS AND WEAKNESSES:\nThe motivation of leveraging equivariance of  monocule RGB image is very good, with potential impact to the field. This setting will facilitate policy learning with broader applications. The experiments are generally good and results are promising.\n\nWeakness:\n(1) Sim2Real Transfer and Robustness: There is limited discussion/experimental evidence on how robust the representation is under severe domain shift, eg, lighting changes, partial occlusions, or unexpected camera movement/perturbations. Most real-world tasks could be regarded as within well-prepared settings.\n(2) Computational Efficiency Analysis: The supplementary mentions that equivariant networks increase training time. However, the main paper lacks quantitative analysis of inference or training speed relative to baselines, and does not address potential impacts on deployment in real-time applications.\n(3) Requirement for Segmentation. Many recent equivariant robotic manipulation models exhibit strong robustness to unsegmented inputs. In contrast, this model relies on object segmentation. While segmenting target objects is not particularly difficult these days due to open-vocab segmentation models like SAM, this requirement restricts model’s applicability to object-centric tasks. With segmentation, the model is unable to manipulate non-object entities (e.g., a pile of tiny objects as a whole) or understand global scene context.\n(4) Failure Cases and Limits: Are there specific types of manipulation (e.g., heavy occlusion, transparent objects) where the method's SO(3)-equivariance leads to failure or decreased performance?\n\nLimitation:\nThe authors provide a clear limitations section, candidly discussing the restrictions in camera, lack of translational equivariance, single-arm policy scope, and compute cost. Suggestions for improvement (e.g., incorporating more failure cases study) are constructive.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes. The authors provide a clear limitations section, candidly discussing the restrictions in camera, lack of translational equivariance, single-arm policy scope, and compute cost.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}