Title: MonoLift: Learning 3D Manipulation Policies from Monocular RGB via Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Robotic Manipulation

Cross-Modal Knowledge Distillation

Policy Learning
Abstract: Although learning 3D manipulation policies from monocular RGB images is lightweight and deployment-friendly, the lack of structural information often leads to inaccurate action estimation. While explicit 3D inputs can mitigate this issue, they typically require additional sensors and introduce data acquisition overhead. An intuitive alternative is to incorporate a pre-trained depth estimator; however, this often incurs substantial inference-time cost. To address this, we propose MonoLift, a tri-level knowledge distillation framework that transfers spatial, temporal, and action-level knowledge from a depth-guided teacher to a monocular RGB student. By jointly distilling geometry-aware features, temporal dynamics, and policy behaviors during training, MonoLift enables the student model to perform 3D-aware reasoning and precise control at deployment using only monocular RGB input.
Extensive experiments on both simulated and real-world manipulation tasks show that MonoLift not only outperforms existing monocular approaches but even surpasses several methods that rely on explicit 3D input, offering a resource-efficient and effective solution for vision-based robotic control. The video demonstration is available on our project page: https://robotasy.github.io/MonoLift/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MonoLift, a tri-level knowledge distillation framework that enables learning 3D manipulation policies from monocular RGB images by transferring spatial, temporal, and action-level knowledge from a depth-guided teacher to an RGB-only student model.\n\nSTRENGTHS AND WEAKNESSES:\n1.\tWhile the paper claims to eliminate inference-time overhead, it still requires access to high-quality depth estimation during training. This dependency on pre-trained depth estimators may limit the method's applicability in scenarios where obtaining reliable depth estimates during training is challenging or expensive, somewhat undermining the core motivation of making 3D manipulation learning more accessible.\n2.\tThe paper lacks theoretical analysis of what specific 3D knowledge is being transferred through the tri-level distillation and why this particular combination of spatial, temporal, and action distillation is optimal. While the empirical results are promising, the absence of theoretical justification makes it difficult to understand the fundamental principles and predict when the approach might fail.\n3.\tAlthough the results are consistent across tasks, the absolute performance gains are often modest. For instance, on Meta-World, MonoLift achieves 87.8% vs 83.2% for the best 3D baseline - a meaningful but not transformative improvement. The gains over RGB-only methods are more substantial but vary significantly across tasks, suggesting limited robustness.\n4.\tThe demonstrated tasks, while spanning multiple benchmarks, are primarily focused on relatively standard manipulation scenarios. The evaluation lacks more complex manipulation tasks that would truly validate the claimed advantages of 3D-aware reasoning, and the real-world experiments are limited to basic manipulation primitives.\n5.\tWhile the tri-level distillation framework is well-designed, the individual components largely build upon existing techniques (standard depth estimation, transformer architectures, knowledge distillation). The primary contribution is the integration and application to manipulation learning rather than fundamental algorithmic innovations.\n\nQUESTIONS:\nPlease refer to my comments above.\n\nLIMITATIONS:\nPlease refer to my comments above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MonoLift, a policy-distillation framework that applies tri-level knowledge distillation: (1) Spatial Distillation (matching the student’s RGB-only features to the teacher’s depth-enhanced features), (2) Dynamic Distillation (aligning temporal feature gradients), and (3) Action Distillation (minimising the KL divergence between student and teacher action distributions). The authors present an extensive experimental evaluation in both simulation and the real world, showing that MonoLift consistently outperforms a broad set of baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The paper is well-motivated and timely: RGB-only perception is increasingly popular, and the proposed approach retains depth-related spatial cues during training while keeping test-time inference lightweight.\n2. The experimental design is thorough, spanning several environments and a wide array of baselines. The empirical results are convincing.\n\nWeakness\n\nMy main concern is the absence (or, at least, the lack of clarity) of a baseline that passes estimated depth to the policy at inference time instead of relying on distillation. The introduction claims that attaching a depth estimator `significantly increases computational cost and latency`, which is the core motivation for spatial distillation. Yet it is unclear whether such an RGB + Depth baseline is actually evaluated. Table 2 seems intended to quantify this cost, but it is not clear from lines 358–368 what this baseline contains. Moreover, including a clearly defined RGB + Depth baseline in the main quantitative results (Table 1 and Fig. 3a) would also let readers weigh accuracy against compute.\n\nQUESTIONS:\n1. As mentioned in Weakness, what precisely is the “RGB + Depth” baseline in Table 2? Does it use the same architecture as MonoLift but with RGB + estimated depth inputs?\n2. Using a small MLP as the policy head feels limiting, especially given the recent advances with ACT and Diffusion Policy heads. Implementing MonoLift with one of these stronger heads might further raise performance.\n3. In the ablation study, did the authors remove the corresponding distillation loss in the Student for each of the ablated variations? \n4. There is a typo `$M$ linear layers` in line 218.\n5. A video showcasing the real-world experiments would help readers understand qualitative performance.\n6. Classical distillation schemes first train (and freeze) the teacher before optimizing the student. Did the authors try this sequential alternative? What advantages led you to prefer the end-to-end joint approach reported in the paper?\n\nLIMITATIONS:\nAlthough the authors discuss limitations, the section is relegated to the appendix. Moving it into the main text in the final version would improve transparency and balance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
In this paper, the authors propose a distillation framework for manipulation policy learning, where a teacher model that incorporates estimated depth, temporal dynamics is used; and a student model which only uses RGB image is used to distill the policy. The authors conduct extensive experiments on LIBERO-90, Meta-World and LIBERO-LONE. The proposed method outperforms baseline methods on these benchmarkes. Ablation studies are conducted to validate the model design choice.

STRENGTHS AND WEAKNESSES:
Stength:

The proposed method is simple yet effective, which can be promising and valuable for the community. The authors conduct extensive experiments and ablation studies to validate the effectiveness of the method. 

The paper is well written and structurd. Important details are provided for reproducibility.

Weakness:
The effectiveness of the method on challenging tasks, where the depth estimation can be wrong, such as transparent and specular objects, is not shown.

QUESTIONS:
1. What are the weights of the three losses? How are they determined?How sensitive is the method to the weights?

2. More results of the teach models can be added, including performances, learning curves.

3. For the real-world experiments, the baseline performances are unexpected low. More explaination can be added.

LIMITATIONS:
The limitations are discussed in the supp.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MonLift, which learns a single-view manipulation policy with 3D awareness by distilling information from a teacher model during training. The proposed method uses three kinds of knowledge distillation:  spatial representation distillation by aligning feature to a teacher model that encodes both RGB and estimated depth; temporal dynamics distillation by aligning the temporal student feature towards teachers by matching the temporal gradients; the  action distribution distillation that matches the student's action distribution to the teacher's. During deployment, only the student policy with single-view image observation is needed. \n\nExperiments are conducted on Libero and Metaworld benchmarks. The proposed method is shown to outperform several prior methods that use either explicit 3d information at test time or first reconstruct 3d information at test time instead of distilling the 3d knowledge at training time.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- This paper is well-motivated. 3D information is essential for manipulation and how to infer that purely from single view RGB image is important.\n- The paper is generally well-written and easy to follow.\n- The experiments are relatively comprehensive and show good results of the proposed method.\n\nWeakness:\n- I think the paper is missing important comparison to some well-established multi-view RGB imitation learning baselines, such as Diffusion Policy, OpenVLA, and Pi-0. These methods use multi-view RGB images as input and has demonstrated strong performance on many imitation learning benchmarks. E.g. Table 12 in OpenVLA shows that OpenVLA and Diffusion Policy can achieve strong results on the Libero Benchmarkings. It is unclear how the proposed method compares to these stronger baselines, which makes it hard to access the true merits of the proposed method.  \n- Some details and ablations are missing from the paper. See the questions section below.\n\nQUESTIONS:\n- In figure 1 of the paper it is mentioned that  Multi-View RGB requires preprocessing and alignment. What preprocessing and alignment are needed? In my understanding, most modern multi-view RGM imitation learning algorithms do not need any specialized ways of fusing the multiview RGB images -- simply concatenating them channel wise or concatenating the feature vector is enough, e.g., as done in Diffusion Policy, OpenVLA, Pi-0, etc.\n- Regarding Temporal dynamics distillation – an ablation on using l2 loss between teacher and student features instead of gradient matching may help validate this choice.\n- Can the authors add a discussion on why is Action Distribution Distillation is needed if we already have expert action labels and the standard imitation learning loss using the expert action labels? Why additionally imitate the teacher actions would help?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}