DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
Authors: Jingyi Tian
Le Wang
Sanping Zhou
Sen Wang
lijiayi
Gang Hua
Venue: NeurIPS 2025 poster ()
Keywords: embodied AI
robotic manipulation
representation learning
Primary area: reinforcement_learning

Abstract: Learning generalizable robotic manipulation policies remains a key challenge due to the scarcity of diverse real-world training data. While recent approaches have attempted to mitigate this through self-supervised representation learning, most either rely on 2D vision pretraining paradigms such as masked image modeling, which primarily focus on static semantics or scene geometry, or utilize large-scale video prediction models that emphasize 2D dynamics, thus failing to jointly learn the geometry, semantics, and dynamics required for effective manipulation. In this paper, we present DynaRend, a representation learning framework that learns 3D-aware and dynamics-informed triplane features via masked reconstruction and future prediction using differentiable volumetric rendering. By pretraining on multi-view RGB-D video data, DynaRend jointly captures spatial geometry, future dynamics, and task semantics in a unified triplane representation. The learned representations can be effectively transferred to downstream robotic manipulation tasks via action value map prediction. We evaluate DynaRend on two challenging benchmarks, RLBench and Colosseum, as well as in real-world robotic experiments, demonstrating substantial improvements in policy success rate, generalization to environmental perturbations, and real-world applicability across diverse manipulation tasks.

DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
Jingyi Tian'
Le Wang Sanping Zhou' Sen Wang' Jiayi Li Gang Hua?
National Key Laboratory of fluman-Maching Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi' an Jiaotong University Amazon
Abstract
Learning generalizable robotic manipulation policies remains a key challenge due to the scarcity of diverse real-world training data. While recent approaches have tempted to mitigate this through self-supervised representation learning, most either rely 2D vision pretraining paradigms such masked image modeling, which primarily focus on static semantics or scene geometry, or utilize large-scale video prediction models that emphasize 2D dynamics, thus failing jointly learn the ge- ometry, semantics, and dynamics required for effective manipulation. In this paper, we present DynaRend, representation learning framework that learns 3D-aware and dynamics- dynamics-informed triplane features via masked reconstruction and future prediction using differentiable volumetric rendering. By pretraining on multi-view RGB-D video data, DynaRend jointly captures spatial geometry, future dynamics, and task semantics in unified triplane representation. The learned representations can be effectively transferred to downstream robotic manipulation tasks via action value map prediction. We evaluate DynaRend on two challenging benchmarks, RL- Bench and Colosseum, as well as in real-world robotic experiments, demonstrating substantial improvements policy success rate, generalization to environmental perturbations, and real-world applicability across diverse manipulation tasks.
1 Introduction
Developing versatile robotic control policies capable of performing diverse tasks across varying enviromments has emerged as an active area of research in embodied AI [4, 3, 25, 38, 1]. Despite the promise of end-to-end approaches for generalizable robotic control, the lack of abundant, diverse and high-quality robot data remains key bottleneck.
To address this, recent works leverage self- supervupe methods to learn transferable visual repre- sentations for downstream policy learning. One line of research [30, 23, 28, 33] directly adopts 2D vision paradigms such contrastive learning and masked image modeling. While effective capturing high-level semantics, these methods overlook the specific needs of robotic manipulation, including 3D geometry understanding and future dynamics modeling. Another line [17, 41] focuses on predictive representations via future prediction. using large-scale video generative models to learn object and environment dynamics. However, these approaches mainly model dynamics in 2D and lack explicit awareness of the underlying 3D scene structure. A few recent efforts [26, 19] explore 3D dynamics learning with explicit representations like dynamic Gaussians, but these introduce significant structural complexity, limiting flexibility and scalability for downstream policy learning.
"Corresoonding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Current States
Faturo
Prediction
Reconstruction
Volumetric Rendering
(a) Dynamics Representation
(b) Static Representation
(c) Unified 3D Representation
Figure I: Comparison of representation learning paradigms for robot learning. (a) Learning predictive 2D representations [17] /7]b by forecasting future frames from the current observation to capture future dynamics. (b) Learning semantic or geometric features through reconstruction of static scenes using MAE [33] or 3D reconstruction [46]. (c) Our approach leverages differentiable volumetric rendering to jointly learn semantics, geometry, and dynamics in a unified 3D representation.
In this paper, we introduce DynaRend, representation learning framework for robotic manipulation focusing on learning generalizable visual representations via 3D-aware masked future rendering. Our work leverages neural rendering to perform pretraining on multi-view RGB-D video data, enabling model learn representations that are grounded both spatial geometry and future dynamics. Specifically, we begin by reconstructing point cloud from multi-view RGB-D observations, which then projected onto a triplane representation. A random subset of the triplane features masked and replaced with Iearnable embedding. The masked features and the language instruction processed through a reconstructive and a predictive model, yielding two intermediate feature volumes that represent the current supervision, we randomly select one current and one future frame, and extract their semantic features using a pretrained vision foundation model such as DINOv2 [31], J.. We then sample rays and apply differentiable volumetric rendering to generate RGB, semantics, and depth outputs from the predicted volumes, which supervised by ground-truth views without annotations. After pretraining, use the reconstruction and dynamics models to extract triplane features, which are then fine-tuned with an action decoder to predict action value maps on downstream manipulation tasks.
state and the predicted future counterpart, respectively. To provide
Compared to prior approaches, by jointly modeling spatial geometry, future dynamics, and task semantics through rendering-basee based supervision, DynaRend provides unified framework for learning generalizable and scalable 3D representation tailored to manipulation tasks. Additionally, while previous methods [26, 46] incorporate neural rendering auxiliary supervision, they typically require extensive calibrated novel views to serve as supervision, which is feasible in simulation but impractical in real-world settings with limited camera views. To address this, leverage pretrained risual-conditioned generative models to augment target views by synthesizing novel views from existing views, reducing reliance on dense camera setups and enhancing real-world applicability.
We evaluate our method on two challenging robotic manipulation benchmarks. RLBench [21] and Colosseum [32]. Results show that pretraining framework significantly improves manipulation success rates and exhibits strong generalization to unseen environmental perturbations such as object size, color, and lighting. We further validate our method across five real-world tasks, demonstrating effectiveness and adaptability in practical settings. Our contribution can be summarized as follows:
We propose Dynakend, novel representation learning framework that learns generalizable triplane features via masked future rendering for robotic manipulation.
We conduct a systematic study of different pretraining strategies, including reconstruction and prediction objectives, masking strategy and view synthesis, on the effectiveness of downstream policy learning.
We validate our method through extensive experiments in both simulation and the real world, showing consistent improvements over existing approaches.

---

2 Related Work
Representation Learning for Robotics. recent years, the field of computer vision has witnessed growing research focus on self-supervised learning paradigms, where a variety of techniques [7.9, 9, 8, 5,33J have demonstrated remarkable effectiveness in learning transferable representations without supervision. Inspired by this, prior works have applied such techniques to robotics tasks. Notably, some approaches such as VC-1 [28], Voltron [23], and 3D-MVP [33] focus on learning discriminative visual features through masked image modeling. Other lines of work have explored the use of contrastive objectives [30] and distillation [46] for learning generalizable representations. However, existing approaches primarily focus on 21) static pretraining with limited 3D spatial awareness as well as future dynamics understanding, which are essential for manipulation tasks.
Future Prediction for Policy Learning. Existing approaches have also investigated the use of future prediction as an auxiliary objective to facilitate policy learning. GR-I/GR [42, 6] leverage an autoregressive transformer to generate subsequent frames and actions as next token prediction task. SuSIF [2] and UniPi [10] employ a generative diffusion model to predict future images or video and subsequently train an inverse dynamics model conditioned on the generated goal to predict actions. VPP [17] and VidMan [41] exploit video diffusion models pretrained on large-scale datasets to capture future dynamics, which is subsequently utilized to inform action prediction. Nevertheless, existing methods primarily focus pretraining dynamics models using 2D video data, while neglecting 3D dynamics modeling - an essential capability for robotic manipulation tasks that requires reasoning over object and environment interactions in 3D space. In addition, methods such as Imagination Policy [19] and ManiGaussian [26] have explored learning dynamics based on explicit 3D representations. Despite their ability learn spatial and dynamic information through point clouds or 3D) Gaussians, these methods often suffer from limited scalability and are difficult to directly integrate with downstream policy due to representational complexity of explicit 3D structures. In addition, the effectiveness of these methods often hinges on the availability of extensive novel-view supervision, posing significant challenges for scalability and deployment in real-world scenarios.
Neural Rendering. Recent advances in 3D vision, particularly in neural rendering [29], have enabled scene representation through neural radiance fields that are supervised via volume rendering from multi view images. In light of these developments, prior works [18, 48, 44, 39, 20] have employed differentiable neural rendering to facilitate 3D representation learning. Yet, such approaches remain confined to applications in 3D perception and autonomous driving, with limited exploration in interactive robotics scenarios. Some recent efforts have attempted to bridge this gap by applying such techniques to robot learning. For instance, SPA [47] leverages differentiable rendering to pretrain a 2D visual backbone with enhanced 3D spatial awareness; GNFactor [46] distills features from pretrained visual foundation models via volumetric rendering learn voxel-based representations. However, these methods primarily focus on learning 3D consistency in static environments, which limits their effectiveness in robotic manipulation tasks where capturing future dynamics is essential.
3 Methodology
In this section, we present the proposed DynaRend in detail. We begin by formulating the problem in Sec. 3.1. In Sec. 3.2, we describe the process of feature volume extraction and construction with multi-view RGB-D image inputs. Given feature volumes, we leverage differentiable volumetric rendering to learn a reconstructive model and a predictive model for pretraining, detailed in Sec. 3.3. Finally, Sec. 3.4, we explain how the pretrained representations and models are transferred to downstream robotic manipulation tasks. The overall pipeline is illustrated in Fig. 2.
3.1 Problem Definition
Language-conen conditioned robotic manipulation is a fundamental yet challenging task that requires agents to ground natural language instructions into executable actions based on visual observations. Among various paradigms, keyframe-based manipulation has emerged as a popular approach, where the agent is tasked with predicting the next key action state including the end-effector pose and gripper state - given a goal-conditioned instruction and current observations. These predicted keyframes are then exccuted using low-level motion planner to reach the desired target.
3

---

Mudi-Stew RGGBDD
Cumrrn Fiarm
Rendaned Frarm
Densil
Semelizs
Mask
E
Volumeric Rendering
+ Tharslakon
Reconstruatte
Action Decoder
Modt
Predictive Model
Stack blckk Un change black.
Pretrsining
Fineluning
Figure 2: DynaRend framework overview. (a) We reconstruct the point cloud from multi-view RGB-D inputs, encode it with an MLP, and project it onto three orthogonal planes to produce triplane features. (b) We mask subset of the triplane features and sequentially pass through reconstructive network and a predictive network to obtaín current and future scene representations. For pretraining, both triplane volumes are rendered into RGB, depth, and semantic maps via volumetric rendering and supervised by corresponding current and future target views. (c) For finetuning, two networks serve triplane encoder and are trained with action decoder on demonstration data.
One effective way to learn keyframe- based policies is through imitation learning [36, 13, 12], which allows the agent mimic expert behaviors by learning from demonstrations. Each demonstration consists of trajectory sequence where each element is represented as triplet including visual observation O, language instruction I, and end-effector state A. A trajectory's keyframes can be identified via heuristic rules [36]. The learning objective for the agent is to predict the end-effector state of the nearest future keyframe, conditioned on the current observation O and the instruction
3.2 Representing 3D Scenes as Triplanes
Recent advances in 3D representation learning for robotic manipulation have explored various scene encodings, including voxel grids [36, 46], point clouds [24], and 3D Gaussians [26]. While effective in capturing geometric details, these representations are either computationally expensive or structurally complex, making them difficult to scale and generalize across tasks manipulation -ppeeiii domains. To balance efficiency and expressiveness, we adopt triplane representation to encode 3D scenes in compact and learnable manner. Specifically, given the observation consisting of a of calibrated multi-view RGB-D images O 1/,, /2, In). ). we first reconstruct scene-level point cloud using depth mack-projection. The resulting point cloud then encoded through an MLP to extract per-point features. To construct the triplane feature representation, we divide the 3D workspace into regular voxel grid of size x x D. and apply axis aiis-aligned max pooling to project the point features onto three orthogonal planes:
fay R #XWXC E EXXDXC fys R.WWXCC
where C is the feature channel. The resulting triplane features V (fay: faz, fyz serve structured, spatially-aware encoding of the observed scene for the following reconstruction and dynamics modeling in the pretraining stage.
3.3 Rendering with Volumetric Representation
Masked future prediction. After constructing the triplane-based scene encoding, we aim to learn generalizable representations that are both 3D-aware and lynamicc--nfoomel for downstream policy learning. To this end, we formulate the pretraining task combination of two complementary objectives: reconstruction, which encourages understanding of scene geometry, and future prediction, which guides the model to capture future dynamics. Specifically, we first randomly mask subset of the triplane features and replace them with learnable mask embedding, resulting partially masked triplane representation denoted as 1> {Fay, fzz, )- To incorporate task-specific information, we encode the language instruction using a pretrained CLIP [34] text encoder and concatenate resulting embeddings I with the triplane features, which are then fed into a reconstructive network

---

Erecnn to reconstruct the complete 3D feature representation of the current scene:
Vnew
FROW 25
- Erecon (V, 1).
(2)
yz
Following reconstruction, the output feature volume is further processed by predictive network Epred to predict the 3D representation of the nearest future keyframe:
V Vitture
{Fry efuture efuraee } = Epeca (V (Voow> 1).
Both the reconstructive and predictive networks share the same architecture and are jointly used as the triplane encoder for downstream policy learning. Each network adopts a standard Transformer architecture with four layers, enhanced with recent techniques (i.e., SwiGLU [35], QK Norm [16], and RoPE (37J) to enhance stability and expressiveness.
Volumetric rendering. Given the reconstructed current and predicted future triplane features, we apply differentiable volumetric rendering [29] each of them independently, using target views to supervise both the current state reconstruction and future state prediction. During training, we randomly sample subset of pixels from a selected target view for supervision. Each pixel is associated with camera ray r(t) o + td, which can be defined by the camera origin o, view direction d and depth t E [f far) To render pixel, we sample N ray points {p: = o t,dli 1.... N. f < 1 along the ray For each sampled ray point, we project its 3D coordinates onto the three axis-aligned planes (i.e., the x y, x 2, and y/ - = planes) of the triplane features. Bilinear interpolation is then applied on each plane to query features at the projected locations. The features from the three planes are aggregated via summation to obtain a feature descriptor v; for each point along the ray. These pointwise features are subsequently decoded through lightweight MLP heads to predict the attributes for each sampled point: 1) density (v.): RC + R4; 2) RGB values c(v;, d): CC+3 + R3; 3) sematic feature s(v., d): RC+3 + R Following the formulation, the final rendered outputs for each pixel are obtained by integrating the predicted attributes along the ray:
N
Č(r, V)) - Σ w. d), $ Σ wis((:, d), Dir = Σ wilsr
where Č, $ and D are the rendered RGB color, sematic featueres and depth respectively, we T1 (xxeexpp( is the weight for each ray point, 7; = exp(- -j (Vz));)) is the accumulated transmittance, and d; f++1 t is the distance to the previous adjacent point.
Target view augmentation. Existing rendering- based pretraining methods [46, 26] often rely on additional camera viewpoints as supervision, which is feasible in simulation but impractical in real-world setups due to limited camera views. To overcome this, we leverage pretrained generative models to synthesize novel views without extra cameras. Specifically, for a given target frame, we start with a set of calibrated multi-view RGB-D images. We randomly select a base view and perturb its camera pose to define a target view. The multi-view reconstructed point cloud is then warped to the target pose via projection and back-projection. We then employ See3D [27]. pretrained iisual-conditione multi-view diffusion model, to generate realistic images conditioned on the warped
views, and we estimate depth maps from the synthesized views using Depth Anything v2 [45]. These generated RGB-D pairs are used as additional supervision for pretraining. In both simulation and real-world experiment setups, we rely solely on data from fixed camera viewpoints.
Loss functions. During pretraining, we randomly select two target views from the current frame and future frame to supervise the reconstructed and predicted triplane representations, respectively. To improve training efficiency, for each target view, we randomly sample pixels at each iter- ation. The rendering loss is computed as the mean squared error between the rendered outputs and the corresponding ground-truth values from the target images. To further distill high-level semantic information into triplane features, extract semantic features from the target views using RADIOv2.5[15) an agglomerative vision foundation model, to encourage semantic consistency in the learned representations. Since our focus lies in rendering pretraining. we do not explore alternative foundation models. The rendering losses for reconstruction and future prediction are formulated as
Crecon = Ael(C(r) Č(r, Vnow )1l ([r))(r) S(r, Voow)ll + AoSiLog D (r). Voow)):
Epea = A.I|C(r)= - C(r. Visnure Il + X, ([()) $(r, Vinnre) || + A,SiLoog (Dr), D(r, Visuar) (5)
5

---

where C(r), ((r) and (r) denote the ground-truth RGB values, sematic features and depth respec- tively, SiLog scale-invariant log loss [11] optimize depth, and A X₂ and d are weights to balance different losses. The overall objective for pretraining is weighted combination of two loss terms for reconstruction and future prediction respectively:
prettrii = Mecoomfrcco + Apred
where Arecon and Apred are loss weights.
3.4 Predicting Actions for Downstream Tasks
To adapt the pretrained triplane encoder to downstream robotic manipulation tasks, we extend it with an action decoder and fine-tune the entire model using expert demonstrations. Following RVT [13], formulate action prediction as multi-view action value map prediction. Specifically, given current observations, inputs are passed sequentially through the reconstructive network Excon and the predictive network Eped without masking extract spatially and dynamically informed triplane representation. The objective is to predict the next keyframe action A {apuse: Agripper } where apose = { Barass a SE(3) is the end-effector pose, and agripper E {0, 1) is the gripper state.
For translation component agans, we process the extracted triplane features through a convolution layer and an upsampling layer to produce action heatmaps over the three orthogonal planes. The ground-truth action translation is projected onto the three orthogonal planes to generate corresponding target heatmaps, which used supervise the predicted action heatmaps via cross entropy loss. For rotation component ayot and gripper state agripper. we query the triplane features at the predicted action translation position by interpolating the three planes and aggregating the resulting features by summation. The resulting feature is then passed through a lightweight MLP to predict discretized rotation Euler angles and the binary gripper open/close state, both supervised using cross entropy loss with respect to the ground-trud truth labels. The final fine-tuning loss is formulated as
Linnetuee Auaan Ca(gaaas Agras) Apa CE(aaot â + Xgripper CB( (aggppppprrr Agripper)
(7)
where årmes âm and agripper are the predicted action translation, rotation and gripper state respectively.
During inference, the predicted orthogonal plane-wise heatmaps are first broadcast and summed across axes to form a 3D heatmap over the workspace. The spatial location with the highest activation is selected as the predicted position of the end-effector for the next keyframe. This position is then used to query the triplane representation for subsequent rotation and gripper state prediction, following the same decoding procedure as during training.
Experiments
4.1 Simulation Experiments
Environmental setup. We conduct simulation experiments on two challenging robotic manipulation benchmarks: RLBench [21] and Colosseum [32]. All experiments utilize a 7-DoF Franka Emika Panda robot arm equipped with a parallel gripper, mounted on a fixed tabletop setup. The input observation at each time step consists of four calibrated RGB-D -D images captured from the front, left shoulder, right shoulder, and wrist viewpoints of the robot, following prior works [36, 13J. We collect 100 expert demonstrations per task for training on both benchmarks.
For RLBench, we consider two widely adopted evaluation settings: 18-task subset from recent works [36, 13, 12, 33], and 71 7ttk setting covering all executable tasks in the suite. For the latter, we follow SPA [47] to divide the tasks into two groups for evaluation. For both settings, each task evaluated over 25 rollout episodes, and we report the average task success rate. Colosseum [32] is a benchmark for evaluating the generalization capabilities of manipulation policies under 12 types of environmental perturbations across 20 tasks, including changes in object color, texture, size, and lighting. For training, we use the 100 expert demonstrations collected in the unperturbed environment for each task. During test time, for each task, we separately apply each of the 12 perturbation types, and rollout 25 episodes per perturbation. We report the average success rate across each perturbation category assess the robustness of the policy to different types of environmental changes.
Comparisons and baselines. We compare the proposed method against various baselines across both benchmarks. For the RLBench benchmark, compare our method against both different

---

Avg.
Avg. Rankj
Pudh
Sweep Duepon
off
Tum
Closc
S.R.1
Speed
Duttons
Block
Grill
Tup
Drawer
Jar
C2F-ARM-BC (22)
20.1
6.4
72.0
16.0
0.0
20,0
68.0
4.0
24.0
PerAct [36]
49.4
92.8-3.0
74,0-15.0
-2.-=.0
2.4-2.0
51.2+4.7
45.2+4.7
(1R)
62.9
100.0...
72.0gu.0
880
58.025.7
52.022.5
30-MVP [33]
67.5
100
48.0
80.0
96.0
96.0
76.0
3D Deffuser Actor (14)
81.3
23
97.6
a..0g..4
99 20.2
96.0ga
960
RVT2
$3.4
22
20.6
100,0
92.0-2.5
100,
94.0-1.7
99.0-1.7
96.0=2.
100,000.0
DenaRend
X3.2
1.5
19.6
100.0
100.0
93.5 93.5.5.4
100.0
100.0
99.21..
91.22444
Drag
Pet in
Plaze Wase
Screw
Open
Stack Blocks
Stack
Put Cupboud
Insert
Sect
Place
Stick
Sale
Bulb
Drawer
Cups
Pcg
Shups
Cupss
C2FARRMBC
12.0
8,0
20,0
0.3
0.3
0.3
4,0
0.0
Perket [30]
89.6-4.1
84.0+3.5
44847.
17.622.
88.0-5.7 5.7
26.4-3.2
2.+2.0
28004444
5.5e4.1
16.844.7
2.4-1.2
RVT(1))
99.2g1.4
91.223..
91025.
48.0g5.7
11.2_6.9
38.8.a.
364
49.6
11.2ga.
36.0gg.
400g9
3D MVP [3]
100.0
92.0
100.0
60.0
$4.0
40.0
36.0
60.0
20.0
26.0
4.0
3D Duluserr Actor 124]
100.0_0.0
936
824
68.3
47.2
656g
443gga
24
2VT2
99,0=1.7
96002.
950+5.3
58.0
74.0-11.
50.0-2.
690=5.9
66.04..
40.00..
38.0
4.
DynaRend
100.0.0.0
960
984g2.5
39.6
31.2.a.
32.....
87.2
31.2ga.
25.62.
Table I: Evaluation results on 18 RL tasks. Each task is evaluated with 25 rollouts under 5 different seeds. We report the average success rate and standard deviation for all tasks.
policy architectures and pretraining strategies. In the 18-task setting, we primarily evaluate against previous oftte-of-the-art policy models. On the 71-task setting, we mainly adopt the strong RVT- [12] architecture for all methods and replace its Transformer backbone with various pretrained visual models to compare different pretraining strategies, including 2D pretraining methods such as MVP [43] and VC-1 [28], and 3D pretraining methods such as SPA [47]. For the Colosseum benchmark, compare against state-of-the-arr baselines that evaluate policy generalization under different domain shifts. The comparison includes three pretraining- based approaches including MVP [43], VC-1 [28] and 3D-MVP [33], as well as RVT [13] architecture training from scratch.
Implementation details. During both pretraining and fine-tuning stages, we apply SE(3) augmen- tations to the input point clouds, camera poses, and action labels. Specifically, we perform random translations along the x, and axes by up to 0.125 m, and random rotations around the z-axis by up to 45 degrees. The triplane grid is constructed with resolution of 16 x 16 x 16. For pretraining, we train the model for 660k steps, while for fine-tuning on downstream tasks, we train for an additional 330k steps. both stages, we use batch size of 256 and set the initial learning rate to 1 x 10 with cosine decay schedule. Training is conducted using 8 NVIDIA RTX 3090 GPUs.
Results on RLBench. We report the average success rates across the 18 RLBench tasks in Tab. 1. Our method significantly outperforms existing sate-of-the-ar approaches, including RVT-2 [12] and 3D Diffuser Actor [24]. Notably, compared to the baseline RVT [13] model, DynaRend achieves an average success rate improvement of 32.3%. Even relative to RVT-22, a two-stage variant of RVT that incorporates additional refinement, our method still shows noticeable gains. In addition to performance gains, DynaRend also exhibits superior efficiency. Our model achieves the best trade-off' between success rate and inference speed when compared to other baseline methods, demonstrating strong manipulation performance without sacrificing computational efficiency.
We further evaluate the scalability of our ap- proach on the larger 71-task RLBench set- ting, comparing DynaRend against RVT-2 mod-
Method
Group (35)
Group (36)
Avg. S.R.
Iwo-stage
MoCov3 [9]
73.7
54.2
63.99
els with various pretraining methods. The re- sults, summarized in Tab. 2, demonstrate that
MAE [14]
78.3
57.7
68.0
DINOv2 [I]
78.2
56.1
67.1
RynaRend consistently outperforms existing ap- proaches, achieving an average improvement of 881.%% 5%1 over two-stage basclines. These base- lines include both 2D pretraining methods (c.g.,
CLIP [34]
76.8
55.7
66.2
MVP [43]
76.2
56.3
66.2
VC-1 [28]
80.1
55.7
67.9
SPA [47]
80.5
61.2
70.8
single-stage
MVP [43], VC-1 [28]) and 3D pretraining meth- ods (e.g., SPA [47]), all implemented within the two-stage RVT-2 [12] model. Additionally,
RVT RVT
71.9
50.4
61.1
DynaRend
81.4
71.8
76.6
Table 2: Results on 71 RLBench tasks.
DynaRend achieves 25.2% improvement over
the single-stage RVT baseline [13]. Moreover, unlike prior methods that rely on large-scale external pretraining datasets, our method is pretrained solely on task-relevant multi-view RGB-D data without additional external supervision. This highlights the efficiency and task-adaptivennss of DynaRend, making it practical for scalable deployment in real-world setups.
7

---

No
Camera Pase
NO Color
70
Ba Background
50
RD Calor
Ablation
Avg. S.R.(%)
DyuaRend
83.2
2.0
w/o, pretraining
76.7
-6.5
Distracton
Texture
MO
w/o, reconstruction future prediction
78.9
80.7
-2.5
w/o, RGB loss
78.2
-5.0
Table Texture
Textur TextuurMadd Model
PerAct
w/o, sematic loss
w/o. depth loss
80.4
82.0
-2.8
-1.2
R3M
w/o, novel view augmentation
79.8
-3.4
RVT
Light
3D-MVP
Table 3: Ablations. Average success rates are reported over RLBench 18 tasks evaluate the impact of different design choices.
Color
Size
Dynalend
Figure 4: Results on Colosseum.
Results on COLOSSEUM. We present the results on the Colosseum benchmark in Fig. 4. Com- pared to existing 2D pretraining methods, such MVP [43] and R3M [30], as well as 3D pretraining approaches like 3D-MVP [33]. DynaRend achieves consistently higher success rates across most categories of environmental perturbations. Additionally, when compared to the RVT baseline trained from scratch, DynaRend demonstrates significantly greater robustness to various types of environ- mental variations. In particular, our method shows notable improvements in scenarios involving object and environment texture variations. We attribute these gains to the robust spatial and physical priors captured during the 3D-ae masked future rendering pretraining, which enables the policy to better generalize to unseen domain shifts without requiring domain randomization during training.
4.2 Ablation Study& & Analysis
Impact of reconstruction and prediction. We study the effectiveness of pretraining and the contribution of each individual objective, namely reconstruction and future prediction, by selectively enabling them during the pretraining stage. As shown in Tab. 3, we report the average success rates across the 18 RIBeecch tasks under four configurations: (1) training the policy model from scratch without any pretraining: (2) pretraining with only the reconstruction objective; (3) pretraining with only the future prediction objective; and (4) pretraining with both objectives combined. The results indicate that pretraining consistently improves downstream policy performance, showcasing the effectiveness of the proposed pretraining strategy. Among the two objectives, future prediction yields greater gains than reconstruction alone. Importantly, combining both objectives leads to the best performance. highlighting their complementary roles: reconstruction helps the model capture spatial geometry, while future prediction encourages learning of future dynamics critical for manipulation.
Impact of mask ratio. Additionally, we perform an abla- tion study on the effect of the masking ratio applied to the triplane features in Fig. 3. Removing masking entirely or ap- plying an excessively high mask ratio both lead to degraded performance. Introducing moderate level of masking im- proves generalization by preventing overfitting to specific camera views. In contrast, a high mask ratio hampers the model's ability to reconstruct meaningful and coherent 3D representations during pretraining, leading to larger gap between the pretraining and fine-tuning stages.
Average S.R.
82
81
0.0
0.2
0.4
0.6
0.8
Mask Ratio
Figure 3: Ablation on mask ratio.
Impact of different rendering objectives. In Tab. 3, we compare the impact of different rendering objectives used during pretraining on downstream task performance. We evaluate the contribution of each loss term, including RGB reconstruction, semantic alignment, and depth supervision, by selectively enabling them and measuring the resulting average success rate across the 18 RLBench tasks. Our results indicate that both the RGB loss and semantic loss are critical for learning effective 3D representations, directly influencing the quality of the learned features, which in turn affects the performance of downstream policy. The depth supervision also provides moderate improvements,
8

---

though its contribution is less significant. We hypothesize that the triplane representation is derived from point clouds, and thus already encodes explicit 3D information to some extent.
Impact of target view augmentation. We further conduct ablation the target view aug- mentation strategy, as shown in Tab. 3. Incorporating synthetic views during pretraining helps mitigate overfitting the limited camera viewpoints and encourages the model learn invariant and robust 3D representations that better capture spatial geometry. This leads to a noticeable improvement downstream success rates, confirming the benefit of view diversity in supervision.
view-
4.3 Real-world Experiments
Environmental setup. We evalu- ate our method on five real-world robotic manipulation tasks and com- pare it against prior state-of-the-art approach. All experiments are con- ducted using Franka Research 3 robot arm equipped with parallel gripper, mounted on fixed table- top setup. For visual input, we em- ploy two calibrated Orbbec Femto Bolt RGB-D cameras, mounted the left and right of the robot. For each task, we collect 30 expert demonstrations, with spatial con- figurations of objects randomized across episodes. Some tasks include multiple variants involving differ- target objects and instructions. Additionally, during testing, we in- troduce distractor objects into the scenes to further evaluate the robustness of the learned policy under challenging and diverse condi- tions. Detailed descriptions of each task provided in the appendix. We pretrain model on the collected real-world dataset for 30k steps with augmented views and fine-tune it for an additional 10k steps. The training hyperparameters kept consistent with the simulation experiments.
Orobec Femnto Bolt (Right)
(Left)
Research
(a) Real-world Setup
(b) Real-world Tasks
Figure Real-world setup and task examples. We evaluate on five manipulation tasks: Put Item in Drawer, Close Pot, Stack Blocks, Sort Shape, Stack Cups.
Quantitative results. In Tab. 4,
Put Item
Stack Blocks
Sort Shapes
Close
Stack
Avg.
we report the average success rates across the five real-world tasks. The results show that our method con- sistently outperforms prior method. Notably, on tasks involving distrac- objects, RVT-2 struggles dis- tinguish berween different unseen items, leading frequent failure cases. In contrast, our method main- tains robust performance, benefit- ing from the pretrained spatially grounded and semantically coher- ent representations. Additionally, in long-horizon tasks such as Stack Cups, DynaRend achieves sizable performance gains, which we attribute to the joint learning of 3D geometry and future dynamics in pretraining stage, facilitating effective reasoning and manipulation in physically complex scenarios.
Drawer
Pot
Cups
S.R.
3DA [24]
30
45
20
37
RVT [13]
55
RVT-2 [12]
45
10
60
15
37
DynaRend
65
60
35
85
40
S7
Put Item
Stack Blocks*
Sort Shapes*
Close
Stack
Avg.
Drawer*
Pot*
Cups*
S.R.
[24
40
RVT [13]
25
10
RVT-2 [12]
15
10
10
45
0
16
DynaRend 55
55
25
65
25
45
Table 4: Real-world performance. We report the average
success rates over 20 rollouts for each task. * indicates tasks where additional distractors were introduced during testing.

---

Conclusion and Discussion
In this work, we proposed DynaRend, a representation learning framework that unifies spatial geometry, future dynamics, and task semantics through masked reconstruction and future prediction with differentiable volumetric rendering. DynaRend learns transferable 3D-aware and dynamics- informed triplane representations from multi-view RGB-D data, which be effectively adapted to downstream manipulation tasks via action value map prediction. Our extensive experiments on RLBench, Colosseum, and real-world setups demonstrate significant improvements in policy performance and robustness under environmental variations. These results highlight the potential of uunderingbbased future prediction for scalable and generalizable robot learning.
Limitations and future work. Despite the promising results, DynaRend still relies on an ex- ternal low-level motion planner to convert the predicted keyframe actions into executable motion sequences. An important direction for future work is to explore how to directly leverage the triplane representations for action sequence prediction, enabling more integrated and end-to-end rohot control.
Acknowledgments
This work was supported in part by the National Key Research and Development Project under Grant 2024YFB4708 100, National Natural Science Foundation of China under Grants 62088 I02, U24A20325 and 12326608, and Key Research and Development Plan of Shaanxi Province under Grant 2024PT-ZCK-80.
References
[1] Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L.,, Hausman, K., Ichter, B., et al., 2024. x0: A vision-anguagg-acttonn flow model for general robot control, 2024. arXiv preprint arXiv:2410.2416.
[2] Black, K., Nakamoto, M., Atreya, P., Walke, H.R., Finn, C., Kumar, A., Levine, S.. Zero- shot robotic manipulation with pre-trained image-editing diffusion models, in: The Twelfth International Conference Learning Representations.
[3] Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T. Driess, D., Dubey, A., Finn, C., al., 2023. Rt-2: Vision-language-attinn models transfer web knowledge to robotic control. arXiv preprint irXiv:2307.15818
[4] Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al., 2022. Rt-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:2212.066
[5] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A., 2021. Emerging properties in self-supervised vision transformers, in: Proceedings the IEEE/CVF international conference on computer vision, pp. 9650-9660.
[6] Cheang, C.L., Chen, G., Jing, Y., Kong, T., Li, H., Li, Y., Liu, Y., Wu, H., Xu, J., Yang, Y., et al., 2024. Gr-2: A generative video- caaannggag-aatiion model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.061.5.
[7] Chen, T., Kornblith, S., Norouzi, M., Hinton, G., 2020a. A simple framework for contrastive learning of visual representations, in: International conference on machine learning, PmLR. pp. 1597-1607.
[8] Chen, X., Fan, H., Girshick, R., He, K., 2020b. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04299
[9] Chen, X., Xie, S., He, K., 2021. An empirical study of training self-supervised vision trans- formers, in: Proceedings the IEFE/CVF international conference on computer vision, pp. 9640-9649.
10

---

[10] Du, Y. Yang. S., Dai, B., Dai, H., Nachum, O., Tenenbaum. J., Schuurmans, D., Abbeel, P., 2023. Learning universal policies via text-guided video generation. Advances in neural information processing systems 36, 9156-9172.
[11] Eigen, D., Puhrsch, C., Fergus, R., 2014. Depth map prediction from single image using multi-scale deep network. Advances neural information processing systems 27.
[12] Goyal, A., Blukis, V., Xu, J., Guo, Y., Chao, Y.W., Fox, D., 2024. Rvt-2: Learning precise manipulation from few demonstrations. arXiv preprint arXivv2406.08555.
[13] Goyal, A., Xu, J., Guo, Y., Blukis, V., Chao, Y.W., Fox, D., 2023. Rvt: Robotic view transformer for object manipulation, in: Conference on Robot Learning, PMLR. pp. 694-710.
[14] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R., 2022. Masked autoencoders are scalable vision learners, in: Proceedings of the IEEE/CVT conference on computer vision and pattern recognition, pp. 16000-16009.
[15] Heinrich, G., Ranzinger, M., Hongxu, Y., Lu, Y., Kautz, J., Tao, A., Catanzaro, B., Molchanov, P., 2025. Radiov2. 5: Improved baselines for agglomerative vision foundation models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6.
[16] Henry, A., Dachapally, P.R., Pawar, S., Chen, Y., 2020. Query-key normalization for transform- ers. arXiv preprint arXiv:2010004245
[17] Hu, Y., Guo, Y., Wang, P., Chen, X., Wang, Y.J., Zhang, J., Sreenath, K., Lu, C., Chen, 2024. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412.1 14803
[18] Huang, D., Peng, S., He, T., Yang. H., Zhou, X., Ouyang, W., 2023, Ponder: Point cloud pre-training via neural rendering, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16089-16098.
[19] Huang, H., Schmeckpeper, K., Wang, D., Biza, O., Qian, Y., Liu, H., Jia, M., Platt, R., Walters, R., Imagination policy: Using generative point cloud models for learning manipulation policies, in: 8th Annual Conference Robot Learning.
[20] Irshad, M.Z., Zakharov, S., Guizilini, V., Gaidon, A., Kira, Z., Ambrus, R., 2024. Nerf-mae: Masked autoencoders for self-supervised 3d representation learning for neural radiance fields, in: European Conference on Computer Vision, Springer. pp. 434-453.
[21] James, S., Ma, Z., Arrojo, D.R., Davison, A.J., 2020. Rlbench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters 3019-3026.
[22] James, S., Wada, K., Laidlow, T., Davison, A.J., 2022. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13739-13748.
[23] Karamcheti, S., Nair, S., Chen, A.S., Kollar, T., Finn, C., Sadigh, D., Liang, P., 2023. Language- driven representation learning for robotics. arXiv preprint arXiv:2302.12766
[24] Ke, T.W., Gkanatsios, N., Fragkiadaki, K., 2024. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402.
(25) Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., al., 2024. Openvla: An open-source laiion-lngguac--action model. arXiv preprint arXiv:2406.09246.
[26] Lu, G.. Zhang, S., Wang, Z., Liu, C., Lu, J., Tang, Y., 2024. Manigaussian: Dynamic gaussian splatting for multi-task robotic manipulation, in: European Conference on Computer Vision, Springer. pp. 346-366.
[27] Ma, B., Gao, H., Deng, H., Luo, Z.., Huang, T., Tang, L.,, Wang, X., 2024. You see it, you got Learning 3d creation pose-free videos at scale. arXiv preprint arXiv:2412.06699.

---

[28] Majumdar, A., Yadav, K., Arnaud, S., Ma, J.. Chen, C., Silwal, S., Jain, A., Berges, V.P., Wu, T., Vakil, J., et al., 2023. Where are we the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing Systems 36, 655-6777
[29] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng. R., 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications the ACM 65, 99-106.
[30] Nair, S., Rajeswaran, A., Kumar, V., Finn, C., Gupta, A., 2023, R3m: A universal visual representation for robot manipulation, in: Conference on Robot Learning, PMLR. pp. 892-909.
[31] Oquab, M., Darcet, T., Moutakanni, T., Vo, H.. Szafraniec, M., Khalidov, V., Fernandez, P.. Haziza, D., Massa, F., El-Nouby, A., et al., 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193.
[32] Pumacay, W., Singh, I., Duan, J., Krishna, R., Thomason, J., Fox, D., 2024. The colosseum:
A benchmark for evaluating generalization for robotic manipulation, in: RSS 2024 Workshop: Data Generation for Robotics.
[33] Qian, S., Mo, K., Blukis, V., Fouhey, D.F., Fox, D., Goyal, A., 2025. 3d-mvp: 3d multiview pretraining for robotic manipulation, in: Proceedings the IEEE/CVF conference on computer vision and pattern recognition.
[34] Radford, A., Kim, J. J.W Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al., 2021. Learning transferable visual models from natural language supervision, in: International conference on machine learning, PmLR. pp. 7748-8766
[35] Shazeer, N., 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202.
[36] Shridhar, M., Manuelli, L.,, Fox, D., 2023. Perceiver-actor: A multi-task transformer for robotic manipulation, in: Conference on Robot Learning, PMLR. pp. 785-799.
[37] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y., 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568, 127063.
[38] Team, O.M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., al., 2024. Octo: An open-source generalist robot policy. arXiv preprint 1221::2405.12213...
[39] Wang. L., Kim, S.W., Yang. J., Yu, C., Ivanovic, B., Waslander, S.. Wang, Y., Fidler, S., Pavone, M., Karkus, P., 2024a. Distillnerf: Perceiving 3d scenes from single-glance images by distilling neural fields and foundation model features. Advances in Neural Information Processing Systems 37, 02334-62366
[40] Wang, P., Bai, S., Tan, S., Wang, S., Fan, 7., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al., 2024b. Qwen2-vl: Enhancing lasion-language model's perception of the world any resolution. arXiv preprint arXiv:2409.1219 121911
[41] Wen, Y., Lin, J., Zhu, Y., Han, J., Xu, H., Zhao, S., Liang, X., 2024. Vidman: Exploiting implicit dynamics from video diffusion model for effective robot manipulation. Advances in Neural Information Processing Systems 37, 41051-41077
[42] Wu, H.. Jing. Y.. Cheang, C., Chen, G.. Xu, J., Li, X., Liu, M.. Li, H., Kong, T., Unleash- ing large-scale video generative pre-training for visual robot manipulation, in: The Twelfth International Conference on Learning Representations.
[43] Xiao, T., Radosavovic, I., Darrell, T., Malik, J., 2022. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173:
[44] Yang. H., Zhang, S., Huang, D., Wu, X., Zhu, H., He, T., Tang, S., Zhao, H., Qiu, Q., Lin, B., et al., 2024a. Unipad: A universal pre-training paradigm for autonomous driving, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15238-15250.
12

---

[45] Yang, L., Kang, B., Huang. Z., Zhao, Z., Xu, X., Feng. J., Zhao, H., 2024b. Depth anything v2. Advances in Neural Information Processing Systems 37, 21875-21911.
[46] Ze, Y., Yan, G., Wu, Y.H., Macaluso, A., Ge, Y. Ye, J., Hansen, N., Li, L.E., Wang, X., 2023. Gnfactor: Multi-task real robot learning with generalizable neural feature fields, in: Conference on Robot Learning. PMLR. pp. 284-301.
[47] Zhu, H., Yang. H., Wang, Y., Yang, J., Wang, L.., He, T., 2024. Spa: 3d spatial-awarenn enables effective embodied representation. arXiv preprint arXiv:2410.08228.
[48] Zhu, H., Yang. H., Wu, X., Huang, D., Zhang, He, X., Zhao, H., Shen, C.,, Qiao, Y., He, T., et al., 2023. Ponderv2: Pave the way for 3d foundation model with a universal pre-training paradigm. arXiv preprint urXiv2230000858.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction are carefully written.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
There is no theoretical result in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experiments can be reproduced based on model descriptions and hyperpa- rameters provided in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: Code will be released in camera-ready version. All of the simulation data used in the paper is public-available
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https:
ccpps.cc/pullic//u guwdee/iee////e CCodeSubeesiaceecceee sssiissiinn wsssonoliccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training details and hyperparameters are attached in the appendix. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due computational resource limitations, we only performed error statistical analysis on some experiments. For other experiments, due to the large amount of test rollouts, the variance between different runs negligible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The corresponding resources are stated in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: pertps:/nneurr http///eurrrppppeeee cc/ppblicc/thhccccuiieeee
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics in the research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper works on academic, publicly-vvallabl benchmarks and is not related to any private or personal data.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose any safety risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the licenses of open-source code, data, and models.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https : http://ies...//eeeeeeeeee http///ris..c//ooeeeeeeeeeeeeeee / for what should or should be described.
20

---

A Implementation Details
Architecture. In this section, detail the architecture of the reconstructive model, predictive model, and render head. Both the reconstructive and predictive models share the same Transformer architecture, which includes recent techniques such as SwiGLU [35], RoPE [37], and QK Norm [] To handle the 3D triplane representations, we utilize rotary position encoding inspired by M-RoPE [40]. The feature dimensions are split into three parts, corre- sponding to the three spatial dimensions of the triplane (ie., y,
Hyperparameter
Value
triplane resolution transformer depth transformer width attention heads ratio render head width render head layers size ray batch size optimizer
16x
768
12
4.0
768
256
and y planes). For each dimension, we apply RoPE encoding separately, which allows the model effectively learn relative positional relationships across the three triplane planes.
32
Adam W
learning rate corase sampled points
0.0001
128
fine depth sampled points
fine uniform sampled points
The render head takes the triplane features, interpolated according to the sampled points, input and outputs the corresponding density, RGB values, and semantic features, The render head is implemented as two MLP layers with residual connections. Each layer consists of a linear layer, followed by layer normalization and ReLU activation followed by linear layer. This architecture enables effi- cient feature processing and representation learning for both the reconstruction and prediction tasks, with the same render head shared across both tasks.
Apred
1.0
Arecon
1.0
Argh
1.0
Asem
Adepth
0.01
Table 5: Hyperparameters.
Additionally, we follow GNFactor [46] by adopting coarse-to-fine hierarchical structure for render- ing. In the "fine" network, we apply depth-guide sampling to refine the predictions, improving the model's s ability reconstruct and predict scene features, especially finer levels of detail.
Target view augmentation. In real-world setups, the limited and often fixed camera viewpoints pose challenge for rendering-based pretraining. To address this lim- itation, we leverage pretrained visual-conditioned multi-view dif- fusion model to generate novel tar- get views as additional supervision. Specifically, utilize See3D [27]. a large-scale pretrained multi-view video diffusion model. Given input RGB-D images for a target frame, first reconstruct the scene's point cloud. We then select an input viewpoint as the base view, from which we sample a new viewpoint with a random angular offset on a sphere surrounding the center of the scene. The angular offset is restricted within a maximum range of +:30 degrees, as we observe that the quality of the synthesized novel view images decreases the offset angle becomes larger. We generate camera trajectory consisting of 25 frames. where each frame corresponds to camera pose interpolated between the original and new viewpoints. We then use point cloud rendering project the recon- structed point cloud onto each frame of the trajectory, producing warped images and corresponding masks. These warped images and masks, along with the original view, are fed into See3ID, which generates high-fidelity novel-view images. Due to the time required for generating these trajectories, we augment only the keyframe viewpoints. For each keyframe, we randomly sample four distinct camera trajectories, generating total of 100 novel view images per keyframe. In both simulation and real-world experiments, we maintain fixed camera setups and viewpoints, with the view augmentation process described above applied consistently.
See3D
Warped Images
Ganerated Images
Camera Trajectory
Figure 6: Target view augmentation pipeline.
Hyperparaamttrs.
We present the hyperparameters used in DynaRend as shown in Tab. 5.
21

---

B Simulation Experiment Details
RLBench-118. For the 18-task setting, follow PerAct [36] and select 18 RLBench tasks that involve at least two or more variations in object types, spatial arrangements, or instructions. This curated subset is designed evaluate the multi-task generalization capabilities of different agents under diverse and realistic conditions. Detailed descriptions of each task can be found in Tab. 6.
Task
Variation Type
# Variations
Avg. Key framees
Language Template
open draver
placement
3
3.0
"open the drawer"
slide block
color
4
4.7
"slide the block to target"
sseep to dustpan
size
2
4,6
"sweep dirt to the dustpan"
neat grill
category
2
"take the off the grill"
turn tap
placement
2
2.0
"tum tap"
put draver
placement
3
12.0
"put the item the drawer" "close the jar"
close jar
color
20
6.0
drag stick
color
the stick drug cube
larget"
stack blocks
color,count
60
14.6
"stack
"screw blocks"
screv
color
light bulli" "put the money away in the safe on the shelf"
put in safe
placement
3
5.0
place sine
placement
"stack the wine bottle
the rack"
put in cupboard
category
9
5.0
"put the
the cupbourd"
sort shape
shape
5
"put the
shape sorter"
push buttons
color
50
3.8
"push the button. (then the button]"
insert peg
color
20
5,0
"put the ring um the spoke"
cups
color
20
10.0
"stack the other cups top of the cup" "place cups on the cup holder"
place cups
count
3
11.5
Table 6: Details on 18 RLBench tasks.
RLBench-71. For large-scale multi-task evaluation, we follow SPA [47] and divide all executable tasks in RLBench into two groups, consisting of 35 and 36 tasks respectively. For each group, train a language-conditing same two-stage architecture, differing only in the choice of pretrained vision encoder. All other components and training hyperparameters remain consistent with the original RVT-2 setup. Reported results are taken directly from SPA [47].
multi-task agent. All RVT-2 [12] baselines are implemented using the
The 35 tasks in Group include: basketball in hoop, put rubbish in bin, meat off grill, meat on grill, slide block to target, reach and drag, take frame off hanger, water plants, hang frame on hanger, wipe desk, stack blocks, reach target, push button, 1amp on, toilet seat down, close laptop 1id, open box, open drawer, pick up cup, turn tap, take usb out of computer, play jenga, insert onto square peg, take umbrella out of umbrella stand, insert usb in computer, straighten rope, turn oven on, change clock, close mi crowave, close fridge, close grill, open grill, unplug charger, press switch, take money out safe.
The 36 tasks in Group 2 include: The 36 tasks in Group 2 include: change channel, tv on, push buttons, stack wine, scoop with spatula, place hanger rack, move hanger, sweep to dustpan, take plate off colored dish rack, screw nail, take shoes out of box, slide cabinet open and place cups. lamp off, pick and lift, take lid off saucepan, close drawer, close box, phone on base, toilet seat up, put books on bookshelf, beat the buzz, stack cups, put knife chopping board, place shape in shape sorter, take toilet roll off stand, put umbrella in umbrella stand, setup checkers, open window, open wine bottle, open microwave, put money in safe, open door, close door, open fridge, open oven, plug charger in power supply.
Colosseum. Colosseum [32] is recently proposed benchmark designed evaluate the gener- alization capabilities of robotic manipulation policies under diverse domain shifts. It consists of 20 manipulation tasks drawn from the RLBench suite, including: basketball in hoop, close box, close laptop 1id, empty dishwasher, get ice from fridge, hockey, meat on grill, move hanger, wipe desk, open drawer, slide block to target, reach and drag, put money in safe, place wine at rack location, insert onto square peg, stack cups, turn oven on, straighten rope, setup chess, scoop with spatula.
Colosseum defines 14 types of perturbation factors to simulate real-world domain shifts. our evaluation, follow the standard setting and apply 12 of these perturbation factors to all 20 tasks
22

---

to systematically assess the robustness and generalization ability of learned policies under unseen conditions. The 12 perturbations can be categorized into three groups:
Manipulation Object (MO) Perturbation: The manipulation object is the item directly manipulated by the robot, such as the wine bottle in place wine at rack location. Manipulation object perturbations include variations color, texture, and size.
Receiver Object (RO) Perturbation: The receiver object is task-relevant but not directly manipulated, such as the rack in the same task. Receiver object perturbations also include changes in color, texture, and size.
These factors affect the overall scene without modifying task- relevant objects. They include changes to light color, table color, table texture, background texture, camera pose, and the addition distractor objects.
Background Perturbation:
Tab. 7 summarizes the applied perturbations for each task. For more implementation details and perturbation configurations, please refer to Colosseum [32].
Variation
MO
RO
MO Color
MO Size
MO Texture
RO Color
RO Size
RO Texture
Object Mass
discrete
continuous
discrate
discrete
comtinuous
discrete
basketball in hoop
ball
hoop
[0.75,1.25]
213
20
[0.75,1.15]
close box
bus
20
10.
close laptop lid
[0.75.1.00]
empty disbwasher
dishwasher
plate
[0.80.1.00]
[0.80.1.00]
213
get ice from tridge
cup
fridge
[0.75.1.25]
213
20
[0.75,1.00) 00]
bockey
stick
ball
20
10.55.1.05]
20
[0.75,1.251
213
[0.550.5]
meat grill
mest
grill
10.65,1.15]
20
hanger
hanger
pole
wipe desk
sponge
beans
[0.75,1.25]
213
20
[1.0.50]]
open drawer
drawer
20
[0.75,1000]
slide block larget
block
213
[1.0,150]
reach drag
block
[0.80.1.10]
10.50,1.00]
[0.5.2255
put money in safe
money
safe
[0.50,1.00]
213
20
213
place wine rack location
bottle
shelve
20
10.85.1.15]
20
[0.85,1151
213
onto square pog
pog
spokes
[1.00.1.50]
10.85,1.151
213
cups
cups
10.75.1.255
tum oven on
knobs
[0.50,1.50]
straighten
rupe chess
20
213
sctup chess
bourd block
[0.75,1.251
213
20
scoop spatula
spatula
[0.75,1.25]
[0.75,1.50]
213
[1.0,5.0]
Table 7: Summary of tasks and their perturbation factors. For more details, check Colosseum [32].
C Real-world Experiment Details
Hardware Setup. For the hardware setup, we use Franka Research 3 robot equipped with parallel gripper. Visual input is provided by two Orbbec Femto Bolt RGB-D cameras, positioned on either side of the robot arm. The cameras calibrated using the kalibr package to determine the extrinsics between them, while easy handeye package is used to calibrate the extrinsics between the camera and the robot frame. The cameras provide RGB-D images at a resolution of 1280x720 at 30Hz. To prepare the images for model input, we resize the shortest edge to 256 and crop them to resolution of 256x256.
Task
Variation Type
Variations
Avg. Keyframes
Language Template
atack cups
color
5
11.5
"stack cups on cup"
stack blocke
color
5
12.1
"stack blocks on block"
sort shape
placement
5.4
"put yellow circle shape sorter"
close pot put item in drawer
placement
5.7
"put lid on pot"
calegory
4
8.3
"pul in drawer"
Table 8: Details on real-world tasks.
Data Collection. The real-world dataset is collected through human demonstrations. Specifically, for each task and scenario, human demonstrator kinesthetically moves the robot arm to specify a series of end-effector poses. Afterward, the robot arm reset to its initial position, and the demonstrator sequentially moves it to the specified poses. During this process, the camera streams and robot arm states are recorded, including end-effector position, joint positions, and joint velocities. For each task, we collect 30 human demonstrations to train the model.
23

---

Task Name
No. Variatinn
MD Color
MO Toxters
Texan
MO Ston
4O Side
Lighs Colos
Tebtl Cokar
Tible Toxure
Disnuctor
Backgronn
Toxtam
Cament
backetbull in houp
100
96
100
96
100
100
100
100
100
40
100
100
close box
96
84
92
92
88.
88
58
96
close liptup lid
100
100
100
100
96
68
100
96
dishussher
0
0
0
0
0
get ace from fridge
92
58
96
88
80
72
96
100
96
64
92
100
bookey
32
36
36
20
44
32
28
36
20
28
28
mest grill
100
96
96
hanger
wipe desk
blok target
put money
place rack
imsnrt ono square pep
12
stack cups
72
08
36
36
72
carn oven on
76
100
30
92
100
100
strighttss rope
80
64
64
24
60
setup chees
20
0
4
16
16
24
24
scoop with spatula
84
56
96
80
88
96
60
16
83
96
56
88
average
64.2
61.0
53.0
67.6
54.2
54.3
51.5
59.0
63.2
63.6
90.0
64.0
63.4
Table 10: All results on Colosseum benchmark.
Keyframe Discovery. Following the approach of PerAct [36], we use heuristic rules to identify keyframes from the collected demonstrations. A frame is considered keyframe if: 1) The joint velocities are near zero, and 2) The gripper open state not changed.
Execution. For robot control and motion planning, we use the Franka ROS and Movelt with RRT-Connect as default planner.
D Additional Results
Additional baselines. We include additional baselines in Tab. to further compare the performance DynaRend with existing pretraining methods the RLBench- 18 tasks under the in-domain setting, where models are pre- trained directly RLBench-118 data.
Method
Pretrain Data
Avg. S.R.
RVT
None
62.9
MVP [43]
RLBench
64.9
3D-MVP [33]
RLBench
67.5
DynaRend
RLBench
83.2
Detailed results. We present the detailed results all tasks in Tab. 10 and Tab. 11.
Table 9: Results 18 RLBench tasks.
E
Visualizations
In Fig. 8 and Fig. 10, we present the results of target view augmentation in both the simulator and real-world environments. Fig. and Fig. showcase the rendering results in the simulator and real-world settings, respectively.
F Additional Qualitative Analysis
In the supplementary materials, we provide episode rollouts of several representative tasks in simulator and wolld demos in the attached videos.
24

---

Method
McCov3
DynaRend
Group 1
basket ball in hin
100
100
100
100
100
100
100
pur
uxat off grill
100
100
100
100
100
100
100
100
100
100
100
grill
76
76
target
reach and drog
100
100
96
88
100
96
100
96
100
100
100
take frame off hanger
88
92
88
84
85
96
plants
hong frame on hanger
24
wipe desk
blocks
reach target
96
60
96
88
100
96
80
92
96
92
100
buttoon
100
100
100
100
100
100
lamp
toilct scat down
100
100
100
100
100
100
100
96
96
100
96
close Tuptop lid
96
96
96
open
pick cup
88
92
92
88
96
96
88
96
96
96
96
54
96
100
computer
play jonga
100
96
96
100
96
100
96
96
100
insert square
28
80
92
umbrella
umbrella
insert ush in computer
12
20
24
24
16
12
straighten rope
56
72
72
56
tum
change clock
64
48
68
close microwave
100
100
100
100
100
100
100
100)
fridge
close grill
92
96
96
96
96
96
96
100
100
96
96
open grill
76
100
100
100
100
100
unplug charger
press switch
84
92
92
88
72
76
84
76
88
92
76
take money safe
100
100
100
100
100
100
Group 2
change channel
0
4
100
100
push buttons
stack wine
12
12
16
40
12
28
28
20
scoop with spatula
place banger on rack
move hanger
72
O
D
D
88
sweep dustpan
take plati off colored dish rack
96
screw nail
32
$2
36
36
36
36
52
32
32
48
56
take shoes of hox
24
36
cabinct open and place cups
lamp off
96
100
96
96
100
96
96
100
100
100
100
pick and
92
take saucepan
100
100
close drawer
100
100
100
100
100
96
100
100
100
100
100
close box
92
96
96
100
phone
toilet scat up
72
80
88
100
88
88
80
92
96
24
books bookshell
28
12
24
28
stack cups
24
40
56
$2
52
48
56
64
68
64
68
knife chopping board
68
72
76
72
place shape shape socter
take toiler roll off stand
100
100
92
76
96
92
88
84
92
96
84
umbrella umbrella
sctup cheekers
68
open window
92
96
96
100
100
96
100
96
100
100
92
open wine bottle
88
100
open microwave
88
put money in safe
100
96
100
88
92
100
96
100
100
100
96
door
100
96
96
open fridge
24
44
52
48
44
36
64
52
32
64
68
12
plug charger power supply
Table 11: All results on 71 RLBench tasks.
25

---

GT
Rendered
Figure 7: Visualization of rendered results in simulation.
warped views
S penerrated views
Reference Frame
warped views generated views
Reference Frame
S warped views
Reference Frame
generated views
Figure 8: Visualization of target view synthesis in simulation.
26

---

C
Figure 9: Visualization of rendered results in the real world.
warped views
Reference Frame
generated views
warped views
Reference Frame
generated views
warped views
Reference Frame
generated views
Figure 10: Visualization of target view synthesis in the real world.
27


Reviews:
[{"review_id": "0967iuoJRQ", "reviewer": "Reviewer_KFSY", "summary": "This paper presents an approach to build a representation which captures both the 3D nature of RGB-D cameras (3D-aware) and the dynamics associated to a robotic manipulation task (dynamics-informed). The representation is also used to solve robotics manipulation tasks by learning (from demonstration data) an action decoder. The approach is validated in two different benchmarks (RLBench and Colosseum) and on real robot experiments. Additionally authors propose some ablation studies.", "questions": "The DynaRend framework reconstructs a scene-level point cloud from a set of  calibrated multi-view RGB-D images. How can this approach be adapted for common robotic use cases involving moving cameras, such as wrist-mounted or head-mounted cameras, where the camera poses are not fixed and change dynamically with the robot's movement?\n\nThe model predicts the end-effector pose and gripper state but does not explicitly represent the robot arm's geometry in its triplane features. How does the policy account for the arm's physical presence to avoid collisions with the environment, especially in cluttered scenes where the arm's configuration is a significant constraint?\n\nPlease clarify the use of DINOv2 which is mentioned at the beginning of the paper (in the context of enhancing the semantics of the representation) and nowhere else in the main text.\n\nThere appears to be a discrepancy in how language instructions are incorporated. The text states that language embeddings from a CLIP text encoder are concatenated with the triplane features and fed into both the reconstructive and predictive networks during pretraining. However, Figure 2 seems to depict the language instruction being used only in the action decoder. Could you please clarify the precise stage and mechanism through which language conditioning is integrated into the model architecture?", "limitations": "The model does not explicitly represent the robot arm's geometry or plan its path to avoid collisions. The responsibility for generating a collision-free movement is not addressed in the paper. While the model may implicitly learn about the scene's geometry, it does not directly output a motion that considers the arm's configuration in cluttered spaces, which could lead to failures if the external planner cannot find a valid path.\n\nNo relevant negative societal impacts noticed by the reviewer.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The paper proposes a very interesting approach which extends neural rendering (i.e. representations suitable to capture 3D scenes) to the context of robot manipulation tasks where both environment dynamics and object semantics become extremely relevant if not essential. Authors proposed a very sound evaluation in simulation with comparison against multiple alternative approaches. The real-robot experiments are instead quite limited and only one alternative approach is presented. Despite the paper being well written, there are some inconsistencies that the authors should address before publication.", "was_revised": true, "final_justification": "Thanks for your comments and the discussion points raised in the rebuttal. Unfortunately, this discussion doesn't change my rating since the rating was mostly motivated by considerations that authors acknowledged to be a limitation of their paper (e.g. Explicit geometry and collision avoidance).", "rebuttal": "Dear Reviewer KFSY,\n\nMany thanks for your valuable comments and questions. We hereby address your concerns as follows.\n\n---\n\n> ### \\[W1\\]: Limited real-robot experiments.\n\nWe appreciate the reviewer’s feedback on the real-robot evaluation. Following the suggestion, we have conducted additional real-world experiments and expanded the set of baseline methods for comparison, including RVT, 3D Diffuser Actor and an ablated variant of our method without pretraining. To ensure a fair comparison, all methods were trained on the same set of demonstration data and evaluated under the same hardware setup. As shown in the following table, our approach achieves a 20% absolute improvement in average success rate over the strongest baseline and significantly outperforms the no-pretraining variant by 19%, highlighting the critical role of the pretraining in capturing 3D geometry and dynamics for real-world manipulation. Notably, our method shows particularly strong performance in the presence of unseen distractor objects, where it maintains high success rates while other methods degrade substantially, demonstrating its robustness to real-world scene variations.\n\n|  | Put Item in Drawer | Stack Blocks | Sort Shape | Close Pot | Stack Cups | Avg.S.R. $\\uparrow$ |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: |\n| 3DA | 40 | 50 | 30 | 45 | 20 | 37 |\n| RVT | 25 | 40 | 5 | 55 | 5 | 26 |\n| RVT-2 | 45 | 60 | 10 | 60 | 15 | 37 |\n| DynaRend | 65 | 60 | 35 | 85 | 40 | **57** |\n| DynaRend w/o pretrain | 55 | 55 | 30 | 60 | 40 | 48 |\n\nTable: Real-world performance.\n\n|  | Put Item in Drawer | Stack Blocks | Sort Shape | Close Pot | Stack Cups | Avg.S.R. $\\uparrow$ |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: |\n| 3DA | 20 | 15 | 25 | 40 | 0 | 20 |\n| RVT | 5 | 15 | 0 | 25 | 5 | 10 |\n| RVT-2 | 15 | 10 | 10 | 45 | 0 | 16 |\n| DynaRend | 55 | 55 | 25 | 65 | 25 | **45** |\n| DynaRend w/o pretrain | 25 | 30 | 25 | 40 | 10 | 26 |\n\nTable: Real-world performance with additional distractors.\n\n\n\n> ### \\[Q1\\]: Moving camera calibration.\n\nThank you for raising this point. While our real-world experiments utilize fixed cameras for simplicity, the proposed DynaRend framework is fully compatible with moving camera setups commonly used in robotics, such as wrist-mounted or head-mounted configurations. In such cases, standard **eye-in-hand calibration** techniques can be applied to estimate the transformation between the camera and the robot link to which it is attached. Given this transformation and the robot’s joint state, the camera pose can be accurately computed at any time. We will clarify this general applicability in the final version to emphasize that the framework is not limited to static-camera deployments.\n\n> ### \\[Q2 & Limitation\\]: Explicit geometry and collision avoidance.\n\nWe appreciate the reviewer’s thoughtful observation. Our current policy predicts only the next best end-effector pose and gripper state via behavior cloning, and does not explicitly encode the full-body geometry of the robot arm or perform collision checking during policy inference. To ensure safe execution in practice, we rely on **sampling-based motion planners** such as RRT* at deployment time to generate collision-free trajectories from the current arm configuration to the predicted end-effector pose.\n\nWhile this strategy works reliably in most scenarios, we acknowledge that it can occasionally fail in cluttered scenes if the motion planner cannot find a valid path within the allocated time. We recognize this as a limitation of the current design. Looking forward, we believe this can be addressed by integrating our method with end-to-end motion generation frameworks[1][2] that account for full-body kinematics, or by incorporating explicit 3D scene geometry representations such as occupancy grids. These directions would enable collision-aware action generation without relying on an external planner, and we will mention them explicitly in the discussion of future work.\n\n> ### \\[Q3\\]: Clarify the use of DINOv2.\n\nThank you for raising this question. In our framework, we leverage features from vision foundation models such as DINOv2 and CLIP as additional supervision signals during rendering-based pretraining, with the goal of **distilling high-level semantic information** into our learned representation. Since the choice of foundation model is not the primary focus of our work, we adopt a simplified approach by using RADIO[3] as the teacher network. RADIO provides a unified distillation interface by aggregating features from **multiple foundation models** (including DINOv2 and CLIP) into a single representation. We will revise the manuscript to make this design choice clearer.\n\n> ### \\[Q4\\]: Clarify how language condition is integrated.\n\nWe apologize for the confusion. As described in the method section, language instructions are incorporated during pretraining by concatenating CLIP-based language embeddings with the triplane features, which are then fed into both the reconstructive and predictive networks. In Figure 2, we omitted the language branch for **visual clarity** and to better highlight the **core architectural components**. We will update the figure in the final version to explicitly depict the language-conditioning pathway.\n\n---\n\nThank you again for your constructive and detailed feedback. We hope our responses have addressed your concerns, and we will incorporate the above clarifications and improvements in the final version. Please let us know if you have further questions — we will be actively available throughout the rebuttal period and would greatly appreciate your reconsideration of our work.\n\n> references:\\\n[1] Visuomotor Policy Learning via Action Diffusion, 2023.\\\n[2] Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation, 2025.\\\n[3] AM-RADIO: Agglomerative Vision Foundation Model - Reduce All Domains Into One, 2024."}, {"review_id": "nGwKEQZanL", "reviewer": "Reviewer_Jtz2", "summary": "The paper introduces DynaRend, a novel representation learning framework designed to improve robotic manipulation policies. DynaRend learns 3D-aware, dynamics-informed features through masked future rendering using differentiable volumetric rendering. The model is pretrained on multi-view RGB-D video data, capturing spatial geometry, task semantics, and future dynamics simultaneously. Experimental results on RLBench, Colosseum benchmarks, and real-world tasks demonstrate that DynaRend achieves notable improvements in success rates, robustness to environmental variations, and adaptability in practical robotic manipulation scenarios.", "questions": "The main question remains: **What exactly is the primary focus of this paper?**\n\nThe writing of the paper is quite confusing. In the introduction and method part, the authors put emphasis on “representation learning” and “pre-training”, such as “we propose DynaRend, a novel representation learning framework …” and “by pretraining on multi-view RGB-D video data, DynaRend jointly captures spatial geometry …”. This leave me the impression that this paper is doing something like MVP that first pre-training a represenation model on in-the-wild video or image data, and then do the fine-tuning on specific task, either by RL finetuning or collecting some demonstrations to do imitation learning.\n\nBut after reading the experiment, it turns out that the paper did not do any pre-training. Its experiment setting is the same as RVT-2, collecting 100 demos from each task from RLBench and train a single multi-task policy. If you look into the paper of RVT-2, they did not mention a single word about “representation learning” or “pre-training”. “Representation Pre-trainig” and “Multi-task Imitation”, they are totally two different things.\n\nClarification is critical to evaluating the true novelty and positioning of this work. And without reasonable clarification, I currently can not recommend this paper for publication.", "limitations": "- **Reliance on External Motion Planner:** DynaRend currently depends on an external low-level motion planner for converting predicted keyframe actions into executable sequences. An end-to-end integration without reliance on external planners would further validate and strengthen the approach.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- **Experimental Improvements:**\n    - DynaRend achieves considerable performance improvements in real-world experiments, especially notable in long-horizon tasks and scenarios with distractors. The novel 3D representation effectively enhances generalization capabilities under environmental perturbations. I am instrested in seeing more analysis and visualizations of failure cases about RVT-2 compared to the proposed method.\n    - The paper presents insightful ablation studies, clearly illustrating the impact of various components, such as masked reconstruction, future prediction, and rendering losses.\n- **Unified Representation Framework:** The proposed unified 3D representation elegantly integrates geometry, dynamics, and semantics into a single triplane feature representation, showcasing a methodological advance in robotic visual representation learning.\n- **Robustness to Perturbations:** Extensive evaluation on the Colosseum benchmark confirms DynaRend’s robustness to diverse environmental perturbations, including variations in object color, texture, and lighting, underscoring its practical applicability.\n\n**Weaknesses**\n\n- **Unclear Motivation and Positioning:** The paper’s motivation and exact purpose are somewhat ambiguous. While the introduction and method emphasize concepts like \"representation learning\" and \"pre-training,\" experimental setups closely align with the approach of RVT-2 rather than MVP. This discrepancy creates confusion regarding the true positioning and contribution of this work. Refer to the question part for more explanation.\n- **Limited Performance Gain Over Closest Baseline (RVT-2):** If we agree with the first point, then we need to mainly compare the method against RVT-2. The proposed method achieves a modest improvement (83.2% vs. 81.4%) over RVT-2. This incremental gain does not appear statistically significant, especially compared to the substantial improvements presented in RVT-2 over previous models (81.4% v.s. 62.9%).\n- **Insufficient Real-world Experimental Evidence:** Despite repeatedly highlighting the framework’s suitability for real-world scenarios beyond simulation, the paper lacks robust experimental evidence to substantiate this claim. Additional pre-training on datasets such as Ego-4D, similar to MVP, would enhance credibility and real-world relevance.\n- **Lack of Detailed Precision Demonstrations:** While RVT-2 explicitly demonstrates precise manipulations (e.g., peg insertion), this paper lacks similarly detailed visual evidence of precision tasks, leaving uncertainty regarding the model's fine-grained manipulation capabilities.", "was_revised": true, "final_justification": "The rebuttal addresses most of my concerns, and I have accordingly raised my rating. However, one key concern remains regarding the evaluation setup. In the context of representation learning, pretraining is typically performed on out-of-domain data — most notably using in-the-wild datasets such as MVP or 3D-MVP. On the other hand, RVT-2 trains solely on in-domain action data, effectively making it a multi-task reinforcement learning policy. This paper proposes an intermediate approach: pretraining on in-domain actionless data followed by fine-tuning on in-domain action data, though it can also be extended to pre-training on out-of-domain data.\nGiven this distinction, the direct comparison with MVP, 3D-MVP, and RVT-2 is not entirely fair, as the pretraining regimes differ significantly across methods.", "rebuttal": "Dear Reviewer Jtz2,\n\nMany thanks for your valuable comments and questions. We hereby address your concerns as follows.\n\n---\n\n> ### \\[W1 & Q1\\]: Unclear motivation and positioning.\n\nWe respectfully believe there may be a misunderstanding regarding the nature of our approach. Our framework is explicitly designed around a two-stage pipeline:\n1. **Pretraining on multi-view robot data without actions**, where we learn a unified 3D representation that captures both spatial geometry and future dynamics via masked reconstruction and future prediction with volumetric rendering.\n2. **Policy fine-tuning with actions**, where the policy network predicts actions from these pretrained triplane features, effectively performing inverse dynamics reasoning.\n\nThis predictive pretraining significantly benefits downstream policy learning, as shown by our ablations (Table 3), where pretraining improves success rate by 8.4% compared to training from scratch.\n\nOur method is fundamentally different from 2D pretraining approaches such as MVP[1] or VPP[2], which operate on images or videos. In contrast, our pretraining learns **3D-aware triplane representations** that explicitly encode scene geometry and dynamics. Moreover, our view augmentation strategy enables rendering-based pretraining even with limited camera viewpoints, making the approach practical in real-world setups. Looking forward, we plan to extend this framework to larger-scale multi-view or robotic datasets such as DROID[3] and ScanNet[4].\n\n> ### \\[Q2\\]: Limited performance gain.\n\nWe note that **RVT-2 employs a two-stage prediction strategy**, which gives it a structural advantage over single-stage baselines like RVT. Our method, while remaining **single-stage**, still **outperforms RVT-2**, showing strong effectiveness despite the absence of this extra stage. Our framework **could incorporate a similar two-stage design** to further improve fine-grained manipulation performance, as confirmed by our additional two-stage experiment in the following table.\n\n|      | Avg. S.R.$\\uparrow$ |\n| :----      | :-------: |\n| RVT        |     62.9      |\n| RVT-2      |     81.4      |\n| DynaRend   |     83.2      |\n| DynaRend-2 |     **87.8**      |\n\nWe also highlight that performance gains on the standard 18-task RLBench benchmark are inherently limited due to the relative simplicity of the tasks and the **potential upper bound** in achievable success rate. Following prior work, we use datasets generated by PerAct, which include only 100 demonstrations per task. This limited coverage makes it difficult to capture all intra-task variations, naturally constraining improvement margins. To better assess generality, we **evaluate our method on all executable RLBench tasks(71 tasks)**, comparing against RVT-2 variants trained with different pretraining strategies. These results (Table 2) show that our method achieves an improvement of 8.2%, confirming the robustness and scalability of our approach beyond the standard benchmark setting.\n\n> ### \\[Q3\\]: Insufficient real-world experimental evidence.\n\nWe would like to clarify that our current real-world experiments already provide strong empirical validation. Specifically, we pretrain on multi-view RGB-D data without actions, then fine-tune on human demonstrations. Compared to RVT-2, our method achieves a 20% absolute performance gain in success rate.\n\nWe also evaluate **robustness under real-world perturbations** by adding unseen distractor objects. Under these conditions, RVT-2 suffers a 57% performance drop, whereas our method shows only a 21% drop, indicating significantly better generalization. This improvement is largely due to our view augmentation strategy, which synthesizes spatially consistent novel views from a small set of fixed cameras, providing diverse and semantically consistent supervision signals during pretraining.\n\nWe perform **additional real-world comparisons** between a policy trained from scratch and one initialized with our pretrained features, keeping the architecture and demonstrations identical. As shown below, the pretrained variant consistently outperforms the scratch-trained variant, particularly in scenarios with unseen distractors, confirming the value of our pretraining pipeline in practical deployment.\n\n|  | Put Item in Drawer | Stack Blocks | Sort Shape | Close Pot | Stack Cups | Avg.S.R. $\\uparrow$ |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: |\n| RVT-2 | 45 | 60 | 10 | 60 | 15 | 37 |\n| DynaRend | 65 | 60 | 35 | 85 | 40 | **57** |\n| DynaRend w/o pretrain | 55 | 55 | 30 | 60 | 40 | 48 |\n\nTable: Real-world performance.\n\n\n|  | Put Item in Drawer | Stack Blocks | Sort Shape | Close Pot | Stack Cups | Avg.S.R. $\\uparrow$ |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: |\n| RVT-2 | 15 | 10 | 10 | 45 | 0 | 16 |\n| DynaRend | 55 | 55 | 25 | 65 | 25 | **45** |\n| DynaRend w/o pretrain | 25 | 30 | 25 | 40 | 10 | 26 |\n\nTable: Real-world performance with additional distractors.\n\n\n> ### \\[Q4\\]: Lack of detailed precision demonstrations.\n\nWe agree that our current work does not emphasize high-precision manipulation demonstrations, as our main objective is to improve generalization across diverse tasks via transferable triplane representations. While RVT-2 achieves high precision partly due to its two-stage strategy, our single-stage framework could be extended with a similar design for fine-grained control if needed. However, we view such precision-focused extensions as complementary to our primary contribution, which lies in representation learning for multi-task policy improvement rather than optimization for specific precision-oriented tasks. To further address this point, we additionally implemented a two-stage prediction variant of our framework and evaluated it on three precision-oriented manipulation tasks. Results show that with the two-stage extension, our method surpasses the previous SOTA by a clear margin, highlighting its potential for high-precision applications.\n\n|  | Place Cups | Sort Shape | Insert Peg |\n|:-| :-: | :-: | :-: |\nRVT         | 4.0 | 36.0 | 11.2 |\nRVT-2       | 38.0 | 35.0 | 40.0 |\nDynaRend    | 25.6 | 44.8 | 31.2 |\nDynaRend-2  | **52.8** | **54.4** | **55.2** |\n\n---\n\nThank you again for your constructive and detailed feedback. We hope our responses have addressed your concerns, and we will incorporate the above clarifications and improvements in the final version. Please let us know if you have further questions — we will be actively available throughout the rebuttal period and would greatly appreciate your reconsideration of our work.\n\n> references:\\\n[1] Masked Visual Pre-training for Motor Control, 2022.\\\n[2] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations, 2025.\\\n[3] DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset, 2024.\\\n[4] Richly-annotated 3D Reconstructions of Indoor Scenes, 2017."}, {"review_id": "wbCRvvlSnT", "reviewer": "Reviewer_xmna", "summary": "This paper presents DynaRend, a representation learning framework for robotic manipulation. By combining masked reconstruction and future prediction via differentiable volumetric rendering, the method learns spatially and temporally aware triplane representations. Extensive experiments on RLBench, Colosseum, and real-world robotic tasks demonstrate that DynaRend consistently outperforms prior approaches in both performance and efficiency.", "questions": "* What is the temporal prediction horizon used for the future keyframe, and how sensitive is the policy performance to this horizon?\n* How does the model perform when view augmentation (See3D) fails to generate accurate depth or semantics?\n* Could the triplane representation be extended to support policy rollout or full trajectory generation, enabling more continuous control?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* the paper presents comprehensive experimental evaluations across simulated (RLBench, Colosseum) and real-world robotic manipulation tasks. Results show consistent improvements over baselines.\n* a detailed ablation study is conducted to analyze the effects of masked reconstruction, future prediction, rendering loss terms, and view synthesis. These analyses clearly support the core design choices and offer valuable insights into the contribution of each component, which I think make a good contribution to the community.\n\nWeakness:\n* While the integration of differentiable neural rendering into robot learning pipelines has been explored in prior work, this paper’s primary contribution appears to be the introduction of future prediction as a supervision signal for learning temporally-aware representations. If this is indeed the central novelty, it would be helpful for the paper to include deeper analysis or justification of how and why future prediction leads to better downstream performance, beyond the current empirical results.\n* The framework still relies on a low-level motion planner to execute predicted keyframes. While this is common in robotic learning pipelines, it limits the method’s potential for fully end-to-end learning and action sequencing.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer xmna,\n\nMany thanks for your valuable comments and questions. We hereby address your concerns as follows.\n\n---\n\n> ### \\[W1\\]: Justification of future prediction.\n\n- The 3D-aware future prediction component in our pretraining is a core element of our method, intended to learn dynamics-informed representations rather than purely static scene encodings. In robotic manipulation, the ability to anticipate future states inherently captures **future dynamics** and the **consequences of interaction**. By predicting such future keyframes during pretraining, we obtain features that support **inverse dynamics reasoning** during downstream policy learning.\n- While prior works such as VPP have shown the value of 2D future prediction for improving policies, our framework extends this to 3D by **jointly modeling scene geometry and temporal evolution** using triplane-based prediction with volumetric rendering. This results in a unified representation that captures both the spatial structure of the scene and how it changes over time. In our ablation (Table 3), including future prediction yields a 4.3% higher success rate compared to static reconstruction alone, confirming its effectiveness.\n\nWe will further elaborate in the final version on how future prediction specifically benefits downstream performance.\n\n> ### \\[W2 & Q3\\]: End-to-end and continuous control.\n\nWe agree that the current implementation executes predicted keyframes via an external motion planner, and thus does not yet achieve fully end-to-end trajectory generation. However, the learned triplane representation itself imposes no such restriction. It can **readily serve as input to end-to-end control frameworks** such as Diffusion Policy[1] or Trajectory Autoregressive Models[2], enabling direct generation of continuous action sequences conditioned on our 3D-aware, dynamics-informed features. We view integrating such models into our framework as a natural next step to achieve continuous, planner-free execution.\n\n> ### \\[Q1\\]: Temporal prediction horizon.\n\nThank you for raising this question. We would like to clarify that we do not define the future keyframe based on a fixed temporal horizon. Instead, following prior work (PerAct), we adopt a heuristic-based segmentation of the trajectory: (1) Joint velocities are near zero, and (2) The gripper’s open/close state remains unchanged. This procedure partitions the end-effector motion into smooth and coherent sub-actions, each representing a motion primitive. This segmentation reduces learning complexity by focusing prediction on **meaningful transitions** rather than low-level time-stepped actions. Moreover, it removes the need to tune a predefined horizon, improving robustness across diverse task durations and speeds.\n\n> ### \\[Q2\\]: Performance when view augmentation fails.\n\nWe would like to clarify that view augmentation (via See3D) is applied only during pretraining to compensate for the limited and fixed viewpoints in real-world data. During downstream fine-tuning and inference, no view synthesis is used. By introducing spatially consistent RGB-D novel views into pretraining, we **enhance viewpoint invariance** and **prevent overfitting** to specific camera configurations. Importantly, this process does not rely on semantic labels; instead, the model distills high-level semantics from pretrained visual foundation models. Our ablations (Table 3) show a 3.4% downstream performance gain with view augmentation.\n\nWe acknowledge that synthesizing high-quality novel views becomes more challenging with large viewpoint shifts. To ensure fidelity, we restrict augmentation to within ±30° on a spherical surface (Supplementary L43). Qualitative examples are provided in Supplementary Fig. C and Fig. E. To quantify view quality:\n\n1. In simulation: We compare 1000 See3D-generated views with ground-truth novel renderings from the simulator, reporting metrics in the following table.\n\n    | Max angle | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ |\n    | :-------: | :--: | :--: | :---: |\n    | ±15°      |   26.58   |   0.842  |   0.155    |\n    | ±30°      |   24.26   |   0.776  |   0.193    |\n    | ±45°      |   20.43   |   0.714  |   0.258    |\n\n2. In real-world: As ground-truth is unavailable, we perform a human evaluation with five independent raters scoring 200 generated views on a 0–4 scale (0: very poor – 4: excellent), obtaining an average score of 3.27, indicating satisfactory perceptual quality.\n\n---\n\nThank you again for your constructive and detailed feedback. We hope our responses have addressed your concerns, and we will incorporate the above clarifications and improvements in the final version. Please let us know if you have further questions — we will be actively available throughout the rebuttal period and would greatly appreciate your reconsideration of our work.\n\n> references:\\\n[1] Visuomotor Policy Learning via Action Diffusion, 2023.\\\n[2] Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation, 2025."}, {"review_id": "7PyGGLERi6", "reviewer": "Reviewer_ovuc", "summary": "DynaRend is a self‑supervised framework that masks and predicts future triplane volumes inside a differentiable renderer, yielding 3D‑ and dynamics‑aware features. When reused as a frozen/finetuned encoder, these features deliver large gains on RLBench and several real‑world manipulation setups.", "questions": "The questions are the same as the weakness outlined above.", "limitations": "yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Comprehensive evaluation across simulation, robustness benchmarks, and five real‑robot tasks with strong absolute gains.\n2. Clear exposition with a step‑by‑step pipeline diagram, well‑structured text, and thorough ablations.\n\nWeakness:\n\n1. Performance doesn’t seem that strong. The gap between the proposed method and other baselines is slight.\n2. I think the pretrained encoder will overfit to the given dataset distribution. I doubt its generalizability in terms of object types or scenes. In robotics, there are many cases where the robot has to address out-of-distribution cases, but this method couldn’t address this case.\n3. This method requires a calibrated multi-view dataset, which is infeasible in most real-world cases. The author said this makes it practical for scalable deployment in real-world setups, but I think the opposite. The usual/practical robot applications come with 1-2 cameras, which is much smaller than the required camera viewpoints in this work. Furthermore, it hasn’t even enabled the robot to work with a single view during downstream deployment.\n4. No action is engaged in the predictive network, which fails to capture a dynamics-aware feature. Even though the current triplane feature is the same, the next keyframe’s triplane feature would be different according to what action the robot takes. But the current objective in the prediction phase does not consider this, which is contradictory to the author’s arguments.\n5. Techniques are incrementally built upon the existing method, such as SNeRL, RVT, etc. I think this method lacks technical novelty.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ovuc,\n\nMany thanks for your valuable comments and questions. We hereby address your concerns as follows.\n\n---\n\n> ### \\[W1\\]: Limited performance gain.\n\n- RVT-2 attains high performance partly due to its **two-stage prediction strategy**, while our current framework adopts a **simpler single-stage design**. For a fair comparison, our results should be viewed against other single-stage baselines. Even so, our method surpasses RVT-2 under comparable conditions, indicating strong effectiveness. Importantly, our architecture is fully **compatible with a two-stage extension** like RVT-2, which could further increase performance if desired. To verify this, we additionally implemented a two-stage variant of our framework and evaluated it on 18 RLBench tasks, where it achieved clear improvements over the previous SOTA.\n\n    |      | Avg. S.R. $\\uparrow$ |\n    | :----      | :-------: |\n    | RVT        |     62.9      |\n    | RVT-2      |     81.4      |\n    | DynaRend   |     83.2      |\n    | DynaRend-2 |     **87.8**      |\n\n- The RLBench 18-task suite contains relatively simple tasks, where many methods already approach saturation, leaving limited headroom for large absolute gains. Following prior work, we train on the PerAct-generated dataset (100 demos/task), which inevitably omits some task variations and constrains achievable performance. To better measure generalization, we evaluate on **all executable RLBench tasks(71 tasks)** and compare against RVT-2 variants with different pretraining strategies. In this broader setting, our method improves success rate by 8.2%, underscoring its robustness beyond the standard benchmark.\n\n> ### \\[W2\\]: Generalizability in out-of-distribution cases.\n\nOverfitting in imitation learning often arises when policies memorize scene layouts or latch onto spurious correlations, especially given noisy action labels. This results in poor transfer to novel conditions. Our approach addresses this by using self-supervised pretraining via 3D future prediction, enabling the model to capture how the environment evolves and to develop dynamics-aware features that support inverse dynamics reasoning. These features mitigate reliance on brittle visual-to-action mappings.\n\nTo evaluate generalization, we adopt the Colosseum[1] benchmark, which is explicitly designed to evaluate robot policies under 12 distinct types of  out-of-distribution environment perturbations, such as object size, unseen distractors, lighting shifts, novel textures and so on. Each perturbation type targets a different failure mode in manipulation policies, making the benchmark a comprehensive proxy for real-world distribution shifts. Across all perturbation types, our method achieves an average success rate improvement of 18.2% over the previous SOTA, 3D-MVP (in Fig. 4).\n\nAdditionally, To further validate robustness in real-world deployment, we conduct experiments under a challenging setting where novel, unseen distractor objects are introduced into the workspace during policy execution. Under this setting, the success rate of RVT-2 drops sharply by 57%, indicating a strong reliance on memorized scene layouts or specific visual cues. In contrast, our policy—initialized with 3D-aware masked future prediction pretraining—shows only a 21% drop in success rate.\n\nThese results confirm stronger robustness to both synthetic and real-world distribution shifts. In future work, we plan to pretrain on large-scale multi-view datasets (e.g., ScanNet[2], DROID[3]) to further expand coverage of real-world variability.\n\n> ### \\[W3\\]: Reliance on extensive camera viewpoints.\n\nWe appreciate the reviewer for the comment but respectfully note that our method does not require many cameras for deployment. In real-world experiments, we use only **two fixed cameras** and outperform RVT-2 under the same configuration. Four cameras are used only in simulation for fair comparison with prior work. Our method also remains effective with single-view setups. As shown below, performance drops only slightly when using a single camera.\n\n| | \\# Cameras | Put Item in Drawer | Stack Blocks | Sort Shape | Close Pot | Stack Cups | Avg.S.R. $\\uparrow$ |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| RVT-2 | 1 | 50 | 60 | 10 | 60 | 5 | 35 |\n| RVT-2 | 2 | 45 | 60 | 10 | 60 | 15 | 37 |\n| DynaRend | 1 | 65 | 55 | 25| 80 | 45 | 54 |\n| DynaRend | 2 | 65 | 60 | 35 | 85 | 40 | **57** |\n\nTable: Real-world performance.\n\n\n| | \\# Cameras | Put Item in Drawer | Stack Blocks | Sort Shape | Close Pot | Stack Cups | Avg.S.R. $\\uparrow$ |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| RVT-2 | 1 | 15 | 5 | 10 | 40 | 5 | 15 |\n| RVT-2 | 2 | 15 | 10 | 10 | 45 | 0 | 16 |\n| DynaRend | 1 | 45 | 50 | 25 | 65 | 25 | 42 |\n| DynaRend | 2 | 55 | 55 | 25 | 65 | 25 | **45** |\n\nTable: Real-world performance with additional distractors.\n\n\nCrucially, our claim of practicality refers specifically to prior rendering-based pretraining methods (e.g., GNFactor), which rely on rendering large amounts of novel views in simulation—a process that is infeasible in the real world. Instead, we leverage a visual-conditioned diffusion model to generate novel views from limited input views, which substantially reduces the need for dense multi-view data and makes rendering-based pretraining applicable in real-world settings.\n\n> ### \\[W4\\]: Action engagement in predictive network.\n\nWhile we do not feed explicit actions into the predictive network, it is designed to learn dynamics-aware features directly from visual evolution. Introducing action inputs during pretraining risks **information leakage**, potentially causing the model to **overfit to specific action-conditioned cues** rather than learning transferable spatial-temporal structure.\n\nRobotic actions are inherently multi-modal; different actions can yield diverse future states. To control this uncertainty, we predict semantically **meaningful keyframes** (following PerAct) rather than fixed-interval future frames. These keyframes correspond to **stable intermediate states** (e.g., pre-grasp and post-grasp), enabling the network to focus on informative state transitions that align with decision points, rather than noisy intermediate frames.\n\n> ### \\[W5\\]: Novelty of DynaRend.\n\n1. DynaRend introduces a new paradigm for rendering-based representation learning in robotic manipulation by **jointly modeling spatial geometry and future dynamics** within an explicit 3D triplane representation. Unlike prior approaches (e.g., SNeRL, RVT) that focus solely on static scene representation, DynaRend **couples masked reconstruction with future prediction through differentiable volumetric rendering**. This 3D-aware future-prediction objective is directly aligned with the demands of manipulation tasks, where reasoning about how the scene will change after an action is as critical as perceiving its current structure.\n\n2. DynaRend integrates visual-conditioned diffusion-based view synthesis (See3D) into the rendering-based pretraining pipeline. This design enables high-quality novel-view generation from **a minimal number of real-world cameras**, eliminating the dependence on dense multi-view rigs or simulator-rendered images that limit prior methods’ applicability outside simulation. By reducing the hardware and data requirements, DynaRend **makes 3D rendering-based pretraining practically deployable** in real-world robotic systems.\n\n3. Extensive evaluations on RLBench, the Colosseum benchmark (12 perturbation types), and six diverse real-world manipulation tasks consistently show that our approach yields **higher success rates and substantially improved robustness** to appearance, geometry, and viewpoint shifts compared to state-of-the-art baselines. Notably, on Colosseum, DynaRend improves over the previous best (3D-MVP) by 18.2%, and in real-world unseen distractor tests, our policy’s performance drops by only 21%, compared to a 57% drop for RVT-2. These results confirm that the combination of **3D-aware masked future prediction** and **diffusion-based view augmentation** provides a meaningful and transferable advance in representation learning for robot manipulation.\n\n---\n\nThank you again for your constructive and detailed feedback. We hope our responses have addressed your concerns, and we will incorporate the above clarifications and improvements in the final version. Please let us know if you have further questions — we will be actively available throughout the rebuttal period and would greatly appreciate your reconsideration of our work.\n\n> references:\\\n[1] THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation, 2024.\\\n[2] DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset, 2024.\\\n[3] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations, 2025."}]

Author rebuttal:

Meta-review: This submission proposes a representation learning method to enhance robotic manipulation, which learns dynamic 3D features through masked future rendering via differentiable volumetric rendering. The method demonstrates better performance in both simulated and real-world scenarios.

Among the reviewers, one casts the highest-confidence vote for acceptance, while two reviewers vote for borderline acceptance. In contrast, one remaining reviewer votes for rejection, due to concerns about the motivation, limited performance gains, and insufficient evidence of real-world validation. However, AC acknowledges that the method is novel and does deliver practical efficacy. Notably, as a leading machine learning conference, NeurIPS prioritizes novelty as a core evaluation criterion.

Ultimately, AC recommend to accept this submission.
Decision: Accept (poster)