Title: DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
Venue: NeurIPS 2025
Accepted: yes
Keywords: embodied AI

robotic manipulation

representation learning
Abstract: Learning generalizable robotic manipulation policies remains a key challenge due to the scarcity of diverse real-world training data. While recent approaches have attempted to mitigate this through self-supervised representation learning, most either rely on 2D vision pretraining paradigms such as masked image modeling, which primarily focus on static semantics or scene geometry, or utilize large-scale video prediction models that emphasize 2D dynamics, thus failing to jointly learn the geometry, semantics, and dynamics required for effective manipulation. In this paper, we present DynaRend, a representation learning framework that learns 3D-aware and dynamics-informed triplane features via masked reconstruction and future prediction using differentiable volumetric rendering. By pretraining on multi-view RGB-D video data, DynaRend jointly captures spatial geometry, future dynamics, and task semantics in a unified triplane representation. The learned representations can be effectively transferred to downstream robotic manipulation tasks via action value map prediction. We evaluate DynaRend on two challenging benchmarks, RLBench and Colosseum, as well as in real-world robotic experiments, demonstrating substantial improvements in policy success rate, generalization to environmental perturbations, and real-world applicability across diverse manipulation tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDynaRend is a self‑supervised framework that masks and predicts future triplane volumes inside a differentiable renderer, yielding 3D‑ and dynamics‑aware features. When reused as a frozen/finetuned encoder, these features deliver large gains on RLBench and several real‑world manipulation setups.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Comprehensive evaluation across simulation, robustness benchmarks, and five real‑robot tasks with strong absolute gains.\n2. Clear exposition with a step‑by‑step pipeline diagram, well‑structured text, and thorough ablations.\n\nWeakness:\n\n1. Performance doesn’t seem that strong. The gap between the proposed method and other baselines is slight.\n2. I think the pretrained encoder will overfit to the given dataset distribution. I doubt its generalizability in terms of object types or scenes. In robotics, there are many cases where the robot has to address out-of-distribution cases, but this method couldn’t address this case.\n3. This method requires a calibrated multi-view dataset, which is infeasible in most real-world cases. The author said this makes it practical for scalable deployment in real-world setups, but I think the opposite. The usual/practical robot applications come with 1-2 cameras, which is much smaller than the required camera viewpoints in this work. Furthermore, it hasn’t even enabled the robot to work with a single view during downstream deployment.\n4. No action is engaged in the predictive network, which fails to capture a dynamics-aware feature. Even though the current triplane feature is the same, the next keyframe’s triplane feature would be different according to what action the robot takes. But the current objective in the prediction phase does not consider this, which is contradictory to the author’s arguments.\n5. Techniques are incrementally built upon the existing method, such as SNeRL, RVT, etc. I think this method lacks technical novelty.\n\nQUESTIONS:\nThe questions are the same as the weakness outlined above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DynaRend, a representation learning framework for robotic manipulation. By combining masked reconstruction and future prediction via differentiable volumetric rendering, the method learns spatially and temporally aware triplane representations. Extensive experiments on RLBench, Colosseum, and real-world robotic tasks demonstrate that DynaRend consistently outperforms prior approaches in both performance and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* the paper presents comprehensive experimental evaluations across simulated (RLBench, Colosseum) and real-world robotic manipulation tasks. Results show consistent improvements over baselines.\n* a detailed ablation study is conducted to analyze the effects of masked reconstruction, future prediction, rendering loss terms, and view synthesis. These analyses clearly support the core design choices and offer valuable insights into the contribution of each component, which I think make a good contribution to the community.\n\nWeakness:\n* While the integration of differentiable neural rendering into robot learning pipelines has been explored in prior work, this paper’s primary contribution appears to be the introduction of future prediction as a supervision signal for learning temporally-aware representations. If this is indeed the central novelty, it would be helpful for the paper to include deeper analysis or justification of how and why future prediction leads to better downstream performance, beyond the current empirical results.\n* The framework still relies on a low-level motion planner to execute predicted keyframes. While this is common in robotic learning pipelines, it limits the method’s potential for fully end-to-end learning and action sequencing.\n\nQUESTIONS:\n* What is the temporal prediction horizon used for the future keyframe, and how sensitive is the policy performance to this horizon?\n* How does the model perform when view augmentation (See3D) fails to generate accurate depth or semantics?\n* Could the triplane representation be extended to support policy rollout or full trajectory generation, enabling more continuous control?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DynaRend, a novel representation learning framework designed to improve robotic manipulation policies. DynaRend learns 3D-aware, dynamics-informed features through masked future rendering using differentiable volumetric rendering. The model is pretrained on multi-view RGB-D video data, capturing spatial geometry, task semantics, and future dynamics simultaneously. Experimental results on RLBench, Colosseum benchmarks, and real-world tasks demonstrate that DynaRend achieves notable improvements in success rates, robustness to environmental variations, and adaptability in practical robotic manipulation scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- **Experimental Improvements:**\n    - DynaRend achieves considerable performance improvements in real-world experiments, especially notable in long-horizon tasks and scenarios with distractors. The novel 3D representation effectively enhances generalization capabilities under environmental perturbations. I am instrested in seeing more analysis and visualizations of failure cases about RVT-2 compared to the proposed method.\n    - The paper presents insightful ablation studies, clearly illustrating the impact of various components, such as masked reconstruction, future prediction, and rendering losses.\n- **Unified Representation Framework:** The proposed unified 3D representation elegantly integrates geometry, dynamics, and semantics into a single triplane feature representation, showcasing a methodological advance in robotic visual representation learning.\n- **Robustness to Perturbations:** Extensive evaluation on the Colosseum benchmark confirms DynaRend’s robustness to diverse environmental perturbations, including variations in object color, texture, and lighting, underscoring its practical applicability.\n\n**Weaknesses**\n\n- **Unclear Motivation and Positioning:** The paper’s motivation and exact purpose are somewhat ambiguous. While the introduction and method emphasize concepts like \"representation learning\" and \"pre-training,\" experimental setups closely align with the approach of RVT-2 rather than MVP. This discrepancy creates confusion regarding the true positioning and contribution of this work. Refer to the question part for more explanation.\n- **Limited Performance Gain Over Closest Baseline (RVT-2):** If we agree with the first point, then we need to mainly compare the method against RVT-2. The proposed method achieves a modest improvement (83.2% vs. 81.4%) over RVT-2. This incremental gain does not appear statistically significant, especially compared to the substantial improvements presented in RVT-2 over previous models (81.4% v.s. 62.9%).\n- **Insufficient Real-world Experimental Evidence:** Despite repeatedly highlighting the framework’s suitability for real-world scenarios beyond simulation, the paper lacks robust experimental evidence to substantiate this claim. Additional pre-training on datasets such as Ego-4D, similar to MVP, would enhance credibility and real-world relevance.\n- **Lack of Detailed Precision Demonstrations:** While RVT-2 explicitly demonstrates precise manipulations (e.g., peg insertion), this paper lacks similarly detailed visual evidence of precision tasks, leaving uncertainty regarding the model's fine-grained manipulation capabilities.\n\nQUESTIONS:\nThe main question remains: **What exactly is the primary focus of this paper?**\n\nThe writing of the paper is quite confusing. In the introduction and method part, the authors put emphasis on “representation learning” and “pre-training”, such as “we propose DynaRend, a novel representation learning framework …” and “by pretraining on multi-view RGB-D video data, DynaRend jointly captures spatial geometry …”. This leave me the impression that this paper is doing something like MVP that first pre-training a represenation model on in-the-wild video or image data, and then do the fine-tuning on specific task, either by RL finetuning or collecting some demonstrations to do imitation learning.\n\nBut after reading the experiment, it turns out that the paper did not do any pre-training. Its experiment setting is the same as RVT-2, collecting 100 demos from each task from RLBench and train a single multi-task policy. If you look into the paper of RVT-2, they did not mention a single word about “representation learning” or “pre-training”. “Representation Pre-trainig” and “Multi-task Imitation”, they are totally two different things.\n\nClarification is critical to evaluating the true novelty and positioning of this work. And without reasonable clarification, I currently can not recommend this paper for publication.\n\nLIMITATIONS:\n- **Reliance on External Motion Planner:** DynaRend currently depends on an external low-level motion planner for converting predicted keyframe actions into executable sequences. An end-to-end integration without reliance on external planners would further validate and strengthen the approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach to build a representation which captures both the 3D nature of RGB-D cameras (3D-aware) and the dynamics associated to a robotic manipulation task (dynamics-informed). The representation is also used to solve robotics manipulation tasks by learning (from demonstration data) an action decoder. The approach is validated in two different benchmarks (RLBench and Colosseum) and on real robot experiments. Additionally authors propose some ablation studies.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper proposes a very interesting approach which extends neural rendering (i.e. representations suitable to capture 3D scenes) to the context of robot manipulation tasks where both environment dynamics and object semantics become extremely relevant if not essential. Authors proposed a very sound evaluation in simulation with comparison against multiple alternative approaches. The real-robot experiments are instead quite limited and only one alternative approach is presented. Despite the paper being well written, there are some inconsistencies that the authors should address before publication.\n\nQUESTIONS:\nThe DynaRend framework reconstructs a scene-level point cloud from a set of  calibrated multi-view RGB-D images. How can this approach be adapted for common robotic use cases involving moving cameras, such as wrist-mounted or head-mounted cameras, where the camera poses are not fixed and change dynamically with the robot's movement?\n\nThe model predicts the end-effector pose and gripper state but does not explicitly represent the robot arm's geometry in its triplane features. How does the policy account for the arm's physical presence to avoid collisions with the environment, especially in cluttered scenes where the arm's configuration is a significant constraint?\n\nPlease clarify the use of DINOv2 which is mentioned at the beginning of the paper (in the context of enhancing the semantics of the representation) and nowhere else in the main text.\n\nThere appears to be a discrepancy in how language instructions are incorporated. The text states that language embeddings from a CLIP text encoder are concatenated with the triplane features and fed into both the reconstructive and predictive networks during pretraining. However, Figure 2 seems to depict the language instruction being used only in the action decoder. Could you please clarify the precise stage and mechanism through which language conditioning is integrated into the model architecture?\n\nLIMITATIONS:\nThe model does not explicitly represent the robot arm's geometry or plan its path to avoid collisions. The responsibility for generating a collision-free movement is not addressed in the paper. While the model may implicitly learn about the scene's geometry, it does not directly output a motion that considers the arm's configuration in cluttered spaces, which could lead to failures if the external planner cannot find a valid path.\n\nNo relevant negative societal impacts noticed by the reviewer.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}