Differentiable Robot Rendering
Authors: Ruoshi Liu
Alper Canberk
Shuran Song
Carl Vondrick
Venue: CoRL 2024 (2024)
Keywords: Robot Representation
Visual Foundation Model
TL;DR: We introduce a differentiable robot rendering method based on deformable Gaussians splattings and show many downstream applications. Abstract:

Abstract: Vision foundation models trained on massive amounts of visual data have shown unprecedented reasoning and planning skills in open-world settings. A key challenge in applying them to robotic tasks is the modality gap between visual data and action data. We introduce differentiable robot rendering, a method allowing the visual appearance of a robot body to be directly differentiable with respect to its control parameters. Our model integrates a kinematics-aware deformable model and Gaussians Splatting and is compatible with any robot form factors and degrees of freedom. We demonstrate its capability and usage in applications including reconstruction of robot poses from images and controlling robots through vision language models. Quantitative and qualitative results show that our differentiable rendering model provides effective gradients for robotic control directly from pixels, setting the foundation for the future applications of vision foundation models in robotics.

Differentiable Robot Rendering
Ruoshi Liu
Alper Canberk
Shuran Song² Carl Vondrick
CColumbia University
Stanford University
drrobot co umbiu. edu
CLIP
,"a doing IxI")
Text-to-Robot Pose with CLIP
P1
Robot & Camera Pose Estimation
P2
Dr. Robot
p.Ox
d
Motion Rotargeting with Point Tracker
PJ_
p
Differentiably Rendered Rabot
Fonward
Robot Control with Generative Video Model
Backward
Posc Conditioned 3D Gaussian
Figure I: We introduce Differentiable Rendering of Robots (Dr. Robot), robot self-model which differentiable from its visual appearance to its control parameters. With it, we can perform control and planning of robot actions through image gradients provided by visual foundation models.
Abstract: Vision foundation models trained on massive amounts of visual data have shown unprecedented reasoning and planning skills in open-world settings. A key challenge in applying them to robotic tasks the modality gap between visual data and action data. We introduce differentiable robot rendering, method allowing the visual appearance of robot body to be directly differentiable with respect to its control parameters. Our model integrates a kinematics-awart de- formable model and Gaussians Splatting and is compatible with any robot form factors and degrees of freedom. We demonstrate its capability and usage in ap- plications including reconstruction of robot poses from images and controlling robots through vision language models. Quantitative and qualitative results show that our differentiable rendering model provides effective gradients for robotic control directly from pixels, setting the foundation for the future applications of vision foundation models in robotics. Interactive demos and additional visualiza- tions are available at: wwwobot.cs.columbia.
Keywords: Robot Representation, Visual Foundation Model
1 Introduction
How to represent robot? Classically, the appearance of robot is modeled by set of geometric primitives (e.g. triangle meshes), and the deformation of its morphology modeled by its kinematic structure. In this paper, we introduce Dr. Robot, representation of robot self-embodiment based on Gaussians Splatting that fully differentiable from its visual appearance its control parameters.
Equal contribution. Order can be listed either way.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

The differentiability of a robot's model [1, 2, 3] is crucial. A differentiable representation allows us to pass the control signal from pixel space to control space through back-propagation. Having a differentiable self-model of the robot allows us to optimize for visual reward and constraints with gradients instead of resorting to gradient- optimization methods such as evolutionary algorithms or reinforcement learning. This is particularly relevant today, given the growing evidence [4. 6] that visual foundation models provide robust and generalizable signals for control.
To obtain high-quality gradients, Dr. Robot tightly integrates three key components:
Gaussians Splatting. The appearance of robot in its canonical pose can be modeled by Gaus- sians Splatting, which models the robot's geometry and texture. Given a viewpoint, an image can be rendered differently through differentiable rasterizer.
Implicit Linear Blend Skinning, We adapt the traditional Linear Blend Skinming (LBS) tech- nique in graphics to work with Gaussian splitting. Combining with differentiable forward kine- matics, the implicit LBS model can project the positions 3D Gaussians to target positions conditioned on target pose.
Condiiondd Appearance Deformation. To model the change of appearance of a robot under various poses, we use a pose-conditioned appearance deformation model to change the spherical harmonics, scale, opacity, covariance matrix of the 3D Gaussians conditioned on a pose.
We validate the effectiveness the gradients produced by our model by performing robot pose reconstruction from in-the-wild videos. Our optimization-base method outperformed the previous state-of-the-art method by large margin. We also showcase various potential applications of Dr. Robot in several tasks, as shown in Fig. 1. By connecting Dr. Robot to various visual foundation models, we can perform planning and control of robot actions through optimization.
The primary contribution of this paper is differentiable robot rendering method for vision-based robot learning. Our experiments demonstrate that our method provides tool for solving complex robotic tasks through visual foundation models. We believe that visual models, such as text- to-image and text-to-video models, continue to grow in spatial and physical reasoning ability, Dr. Robot will serve as the bridge between these foundation models and robotic control.
2 Approach
2.1 Overview and Formulation
Given robot pose p. we aim to model the visual appearance of the robot I from an arbitrary camera perspective as a differentiable function
I= (=x((p))
where π denotes the image formation process under pinhole camera model, which is a standard dif- ferentiable process. the rest of this section, we will discuss how we formulate / as a differentiable function. differentiable rendering model for robots needs have three key properties: full-body differentiabilityy deformability, and efficiency in rendering. We propose model these three prop- erties with forward kinematics (FK), linear blend skinning (LBS), and Gaussians Splatting (GS), respectively, as depicted in Fig. 2.
Forward Kinematics. Given pose p. we represent the position and orientation of joint j with respect to the base joint as the homogeneous matrix (p),, which the product rigid transfor- mations from parent joints along the kinematic chain:
(p) = II (p,) for A,(p.) = R
i-1
of
i
(2)
where A, denotes the coordinate frame of joint with respect to joint 1.

---

P1
Implicit LBS
Appearance Deformation
P2
Forward Kinematics
QQ
p =
pJ
Figure 2: Rendering Pipeline. Our robot model is composed differentiable components. Forward kinematics projects pose vector into skeleton, Implicit LBS projects 3D Gaussians the robot surface, and Appearance Deformation adjusts appearance of 3D Gaussians.
Canonical Gausssians Splatting, A robot in its canonical pose can be treated static 3D scene, which can be modeled by Gaussians Splatting (GS) [7]. Following the notation convention of the original paper [7], the canonical pose robot can be represented as set of 3D Gaussians G with set of means at } where each Gaussian can be expressed as:
gk Nuu,, RSS R²)
where RSST n² denotes the full covariance matrix, R denotes the rotation matrix, and S denotes the scaling matrix. In addition, a 27-dimensional spherical harmonics coefficients Caks and opacity o are used to represent the apperance of the Gaussian during rendering. During optimization, after every number of optimization steps, each Gaussian can split into two to densify the volume, and Gaussians with low opacity are truncated reduce memory consumption.
Implicit Linear Blend Skinning (LBS). Canonical 3D Gaussian splatting model the robot in static pose, but we would like to render the robot at an arbitrary pose Since optimizing separate canonical 3D Gaussian for every pose is unrealistic, we need to construct a geometric deformation function to relocate the Gaussians in 3D space given a pose. Following SMPL [8], we learn linear blend skinning (LBS) for modeling geometric deformations. Classical LBS expresses the
transformation of each vertex on a mesh as linear combination of joint transformations.
Different from [8], the splitting and truncation mechanism of 3D Gaussians during optimization makes it incompatible with the classical LBS, which assumes the input to be a fixed set of vertices. We therefore propose implicil LBS, which accepts an arbitrary 3D coordinate, and outputs set of weights representing the influence of each of the / joint transforms. Specifically, let W R + R be the implicit LBS function, and μ be the position of one of the 3D Gaussians in canonical GS, the geometric deformation function D can be expressed as:
D), Σ W())
Appearance Deformation. So far, we've constructed an implicit LBS function relocate 3D Gaus- sians according robot poses, but 3D Gaussian also contains appearance and shape information in addition its 3D position, namely the rotation matrix Rg. scaling matrix Sg. and spherical har- monics coefficients Csks,k. and opacity oą- Therefore, we additionally learn another appearance deformation function X to predict the change of these parameters conditioned on the canonical and projected positions of 3D Gaussians:
(AR2, ASk, DOs, Caha,k) x) (pR, p)
(5)
Similar to the implicit LBS function, the appearance deformation function is also implicit, allowing them to share the same MLP architecture except for the input and output dimensions.
Optimization. Putting everything together, given the canonical GS g and a robot pose p, the final posed GS GP can be expressed as:
f(p) GP X(D(G.p).p)
(6)
3

---

D-Gaussians
Ours
Ground Truth
D-Gaussians
Ours
Ground Truth
D-Gaussians
Ours
Ground Truth
Figure 3: Visual Quality of Robot Model. Here, we showcase the learned robot model's visual quality by comparing it with results obtained from Deformable Gaussians [9]. Due to the compli- cated kinematic structure of robot, [9] unable to fit the deformation while ours can.
Given a camera viewpoint ı and its corresponding visual observation we minimize the mean- squared error (MSE) between the prediction and observation learn the full robot model:
min 0x.0w.t
= - - (
Test-Time Optimization test time, we perform planning by optimizing robot actions through image gradients for various objectives, such as visual rewards calculated from visual foundation models. Please refer to Eq. 8, 9, 10 for examples of these optimization objectives.
3 Experiments
3.1 Self Model Quality
A robot self model needs to represent the appearance and morphology of the robot accurately. There- fore, in this section, we evaluate the faithfulness of the renderings and geometry when compared with ground truth against several baselines.
Training data. To train our model, we first curate training dataset using Mujoco based on robot's URDF file composed of images of robot under 10k poses, each captured from viewpoints and split them into training and testing sets. We then this data train Dr. Robot by optimizing the objective function in Eq. 7.
Evaluation Metries.
Once training is finished, we evaluate the quality of the model on the testing set. evaluate the appearance with PSNR score between the rendered images and ground truth. For geometry, we use the positions of the target pose Gaussians and evaluate the Chamfer distance between the ground truth pointcloud.
Baselines. We benchmark our results on 4 different baselines. Firstly, we compare against De- formable Gaussians [9], a state-of-the-art method to model deformable objects using GS. Secondly, we include nearest neighbor, where we retrieve the most similar rendering pointclouds from 1000 samples for evaluation. Thirdly, we include random, where we draw random sample from the train- ing dataset for evaluation. Nearest neighbor and random baselines benchmark the difficulty of our tasks. Finally, we also include ablated version of model where the appearance deformation model is disabled as ablation studies which refer to deform.
The quantitative results are shown Table 1, and we also show examples in 3. Foremost, we observe that using LBS as prior for geometric deformation has a significant positive impact on fidelity when compared with Deformable Gaussians, which uses K-plane [10] for modeling deformation. This gap is particularly notable for long robot arms with many chained joints such as the Panda Arm, where efficiently modeling movement in such high degrees through neural network becomes infeasible.
Results.

---

Camara: Azure
Camera Kinect 360
Camera: Resteense
input Image
Reconstructed Pose
Input image
Reconstructed Pose
Impur
Reconstructed Pose
Figure 4:
Robot Pose Estimation from Single Image. From an input image, we perform
optimization to reconstruct the joint angles of the robot and overlay the final rendering of the robot on top of the input image. These results show that accurate robot poses can be reconstructed from only single image through our robot model. This also demonstrated that our robot model provides high-quality gradients for action optimization.
Table 1: Visual and Geometric Quality of Rendering. The table shows PSNR (higher is better) and Chamfer distance (lower better) metrics for different methods and systems.
PSNR (1)
Shadow Hand Unitrce
Unitree H1 Franka Panda Google Robot
ViperX
300
xArm?
Unitree GO1
Unitree Go2
URS
Onrs
31.44
28.31
27.77
32.84
32,601
30.59
13.25
29.74
29.60
32.01
Deforr
29.75
26.95
26.06
27.09
30.89
29.73
31.68
26.95
28.16
26.35
K-plane
Neurest Neighbor (prev)
13.66
13.07
10.19
20.68
18.32
14.13
25.10
10.80
14.87
13:44
Neamest Neighbor
14.81
15.58
12.20
22.40
19.64
16.38
27.12
11.66
15.84
17.61
Random
7.63
10.06
7.33
16.14
14.33
9.06
18.86
6.52
11.01
10.67
Chamfer Distance
Shadow Hand Unitree
Unitree
Pranka Panda Google Robot
ViperX 300
Unittee GO1 Unitree
Ours
03052
0.215
0.241
0.225
0.399
0.067
0.174
0.084
0.207
No Deform
0.054
0.251
0.263
0.315
0,434
0.074
0.190
0.085
0.089
104
K-planc
9.190
95.880
82,210
118.423
341 141,694
60.260
0.215
95.880
15.730
13.335
Nearest Neighbor (prev)
14.242
4.621
2.648
2.081
4,475
2,438
2.502
7.028
Neighbor
0.667
1.677
Random
0.429
45.580
50.872
51.325
17.342
28.374
79.229
3.611
3.591
111.021
3.2 Robot Pose Reconstruction
We evaluate the quality of gradients produced by Dr. Robot practical task robot joint angles estimation from single in-the-wild image. Given monocular RGB image of robot, optimize jointly the robot pose and camera pose 0, to minimize the following objective function,
min CR (p, - - p.O.
Evaluation Protocols. Following prior works [11, 12], we used the Panda-3CAM dataset compos- ing around 10K images with corresponding robot poses created by [11] for evaluation. We evaluate the reconstruction results by calculating the LI distance between the estimated joint angles and the ground truth joint angles. The dataset is composed of three long videos; we, therefore, perform optimization from scratch the first frame of the video and warm start later frames with joint and camera poses from the previous frame as initialization, similar to what's done in both [11, 12].
Baseline. We compare against the state-of-the-art method on this task - -Robopose [12]. Robopose estimates robot poses from single image by a refiner neural network which takes in the image and the current rendering of the robot from the CAD model of the robot and output the delta joint angles.
Results. Quantitatively, we outperforms the previous SOTA method [12] by 32.9%. We perform consistently better across three camera platforms with different fields of view and mounting posi- tions. As shown Fig. 4, our reconstruction is highly accurate and is able to event reconstruct the gripper poses when they are visible which challenging for prior works.
While camera and robot pose are both unknown in the preceding experiments, method allows for reconstruction with known camera robot parameters, using multiple views at once.
5

---

Table 2: Single- View Robot Pose Reconstruction
Dataset
Ours
Ours (std)
Robopose
Robopose (std)
Panda-3cam Azure
9.33
2.65
16.8
3.38
Panda-3cam Kinect360
12.52
3.58
14.37
3.69
Panda-3cam Realsense
17.83
5.5
24.57
3.71
Step 10
"Victory Sign"
"rock on hand"
"thumbs up"
Step 32
Step
Step 145
Step 200
CLIP Loss
"index finger pointing"
"catt
Optimization Iterations
Figure 5: Text-to-Robot Hand Gestures We perform optimization of joint angles of Shadow Hand maximize the CLIP similarity between the rendered image and text prompt. We show the optimization process (left) as well as final outputs of different prompts (right).
3.3 Applications
In this section, we connect Dr. Robot with vision foundation models through image gradients. Different from Sec. 3.2 where we reconstruct robot poses from images/videos, here show how visual foundation models help robot plan future actions by passing gradients from the robot's visual appearance its control parameters.
3.3.1 Text to Robot Pose with CLIP
Contrastive language-mmag models pretrained on internet data have shown remarkable knowledge of open-world images. In this demonstration, we leverage this ability to optimize the similarity of a robot hand to what CLIP judges to be similar to the given text prompt. In particular, let CLIP be denoted by C, which takes an image and a language embedding, returns the content similarity. Since CLIP completely constructed from neural networks, is fully differentiable. Furthermore, let be our differentiable rasterizer. We optimize the joint angles p for the following objective
min CCLP text prompt) = C(text prompt, क (f(p)))
As shown in Fig. 5, with only gradients provided by CLIP, in the 24-dimensional action spaces of Shadow Hand to obtain semantically meaningful hand poses that correspond to given language prompts. These results demonstrate that Dr. Robot can be directly connected with off-the-shelf large-scale vision- languaee models (VLM) through image gradients.
able perform optimization
3.3.2 Text to Action Sequences with Video Model
Recent works [13, 5] have shown that generative video models can be used for planning and guiding robot behavior. Since action data is not in the pre-training of these models, these approaches train their own inverse dynamics models to estimate robot poses in the generated videos. The ability of Dr. Robot reconstruct images and videos in section 3.2 makes it easy drag-and-drop replacement for inverse dynamics model.

---

"Pick up the blue can."
Generated Video
Text-to- Video
Robot
Current Observation
Recovered Robot Actions
"Pick up the red block."
Generated Video
Text-to-Video
Dr. Robot 386
Current Observation
Recovered Robot Actions
Figure Dr. Robot for text-conditioned robot control. We show that robot actions can be extracted from videos predicted by finetuned text-to-video model and be executed for robot control.
To demonstrate this application, we fine-tune Stable Video Diffusion on 100 episodes from the ASU Table Top dataset [14] conditioned on language embeddings of prompts following implementation of [15]. At test time, we are able condition the video model on novel language prompts, and we reconstruct the robot by optimizing the reconstruction loss LRee outlined in section 3.2. We display samples from this process in figure 6.
3.3.3 Visual Motion Retargeting with Point Tracking
Motion retargetting [16, 17] is long-standing problem in robotics that studies transferring motion from one embodiment to another. In prior works, this is typically done through estimating and matching the kinematic structure, e.g., between a humanoid robot and the human body. Here, we explore new way to perform motion retargeting by matching the keypoint trajectories detected by video point trackers [18]. By using the point tracking model P and differentiable renderer π, we can track the initial points of on the robot across time obtain cirr = P(ci, π (PI:T)) e Txx22 Our goal is maximize the similarity between the tracked robot points cir 7 and the goal points ..:T E RTxC2x2 Since these points lack correspondence, we minimize the sum of Chamfer distances across each time step:
C (rack ciy C1rr)
Σ d(c.)))
Σ d(c,, P(ch 1=t
(10)
As shown in figure 7, gradients produced by the sum of Chamfer distance between two sets of point tracks can be back back-propagated through the point tracker, the image, and the robot model to optimize 24/37-dimensional control parameters. Since our method only requires test-tima optimiza- tion, we relax the need for domain-specific training data, which is very hard for traditional motion re-targeting.
4 Related Works
Differentiable Rendering Rendering transforms 3D representations into 2D images, with differen- tiable rendering enabling gradient calculations via images. Research in this domain explores differ- entiable rendering using explicit 3D representations like meshes [19, 20), voxels [21, 22), and point

---

Point Tracker
Point Tracker
Lyhonfer
phanter
chander
chander
Figure 7: Motion Retargeting with Video Point Tracker. We perform optimization on robot action trajectories minimize the chamfer distance between the point tracks in the rendered video and demonstration video, allowing us to transfer motion across the embodiment gap.
clouds [23, 24), often converting these discrete structures into probabilistic distributions to facilitate gradient calculations. Since 2019, neural 3D representations have become prominent, providing fully differentiable frameworks. Key developments include NeRF [25] for volumetric rendering. DeepSDF [26] parameterizing signed distance functions, and Occupancy Networks [27] for volume occupancy modeling, applied in tasks like 3D reconstruction [28, 29, 30]. Recent advancements address the limitations of explicit representations and slow neural rendering, with Gaussian Splat- ting [7] introducing efficient differentiable rasterization using 3D Gaussians. This approach has been extended to model dynamics [9], physics [31, 32], and deformable objects [33].
Visual Foundation Models for Robotics Recent advancements have been made in pre-training perception models within visuomotor policies to develop robust visual representations, focusing on video prediction and contrastive learning to grasp dynamics and causality crucial for physical interactions. Techniques include predicting future events from current observations [34, 35, 36, 37, 38] and improving visual representations robotics through contrastive learning and masked autoencoding [35, 39, 40, 41, 42, 43, 44]. Studies also focus on deriving generalizable reward function from visual pretraining for reinforcement learning [4, 45, 46].
With the rapid advancements in text-to-video generative models [47, 48, 49,50, 51], there is renewed interest in utilizing internet-scale video pretraining for robotics [5]. Some research employs video generative models as world simulators [52, 13, 47], predicting future scenarios conditioned on spe- cific actions. Concurrently, vidco-language models are being used for longer-horizon planning [53, 54, 55]. this paper, we propose an interface between these visual foundation models and the action parameters of robots, allowing planning and control of robot actions through image gradients.
5 Conclusion
In conclusion, our introduction of differentiable robot rendering bridges the modality gap between visual data and robotic action data, facilitating various applications of vision foundation models to robotic tasks. By integrating kinematics-awar deformable model with Gaussian Splatting, ap- proach ensures compatibility across various robot form factors and degrees of freedom. Both quanti- tative and qualitative experiments confirm that our differeeniable rendering model provides efficient and effective gradients for controlling robots directly from pixel data. We believe that Dr. Robot will become an effective tool future applications with vision foundation models in robotics.

---

References
[1] B. Chen, R. Kwiatkowski, C. Vondrick, and H. Lipson. Full-body visual self-modeling of robot morphologies, 2021.
[2] Y. Hu, B. Chen, and H. Lipson. Egocentric visual self-modeling for autonomous robot dynam- ics prediction and adaptation, 2024.
[3] R. Kwiatkowski, Y. Hu, B. Chen, and H. Lipson. On the origins of self-modeling, 2022.
[4] Y. J. Ma, W. Liang, V. Som, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image representations and rewards for robotic control, 2023.
[5] S. Yang, J. Walker, J. Barker-Holder, Y. Du, J. Bruce, A. Barreto, P. Abbeel, and D. Schuur- mans. Video as the new language for real-world decision making, 2024.
[6] W. Huang, Wang, Zhang. Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models, 2023.
[7] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
[8] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. Black. Smpl: skinned multi- person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851-866. 2023.
[9] G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang. 4d gaussian splatting for real-time dynamic scene rendering. 2023.
[10] S. Fridovich-Ken G. Meanti, F. Warburg, Recht, and A. Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance, 2023.
[11] T. Lee, J. Tremblay, To, J. Cheng, T. Mosier, O. Kroemer, Fox, and S. Birchfield. Camera-to-robot pose estimation from single image, 2020.
[12] Y. Labbé, J. Carpentier, M. Aubry, and J. Sivic. Single-view robot pose and joint angle estima- tion via render and compare, 2021.
[13] Y. Du, M. Yang. B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and Abbeel. Learning universal policies via text-guided video generation, 2023.
[14] Y. Zhou, S. Sonawani, M. Phielipp, S. Stepputtis, and H. B. Amor. Modularity through atten- tion: Efficient training and transfer of language-conditioned policies for robot manipulation, 2022.
[15] B. Van Hoorick, R. Wu, E. Ozguroglu, K. Sargent, R. Liu, P. Tokmakov, A. Dave, C. Zheng, and C. Vondrick. Generative camera dolly: Extreme monocular dynamic novel view synthesis. arXiv preprint arXiv:2405. 14868, 2024.
[16] M. Gleicher. Retargetting motion to new characters. In Proceedings of the 25th Annual Conference Computer Graphics and Inteructive Techniques, SIGGRAPH '98, page 33-42, New York, NY, USA, 1998. Association for Computing Machinery. ISBN 0897919998. doi: 10.1145/280814.. 00122880844.280222 URL httht:://ddo wro/ 11t.111.11111888111 2100/////11..2000...... 280820.
[17] T. He, Z. Luo, W. Xiao, Zhang, K. Kitani, C. Liu, and G. Shi. Learning real-time whole-body teleoperation, 2024.
[18] N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker: It is better to track together, 2023.

---

[19] S. Liu. T. Li, W. Chen, and H. Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7708-7711 2019.
[20] H. Kato, Y. Ushiku, and T. Harada. Neural 3d mesh renderer, 2017.
[21] X. Yan, J. Yang, E. Yumer, Y. Guo, and H. Lee. Perspective transformer nets: Learning single- view 3d object reconstruction without 3d supervision, 2017.
[22] Henzler, N. Mitra, and T. Ritschel. Escaping plato's cave: 3d shape from adversarial render- ing, 2021.
[23] W. Yifan, Serena, S. Wu, C. Öztireli, and O. Dorking-Horuun.. Diffcrentiable surface splat- ting for point-basse based geometry processing. ACM Transactions on Graphics, 38(6):1-14, Nov. 2019. ISSN 1557-7368. 15011001145335550933333555 URL http:://pp.oooiiii.oooo org/10. 044553355089 011555888.333552 3356513.
[24] Pistilli, G. Fracastoro, D. Valsesia, and E. Magli. Learning graph-convolutional representa- tions for point cloud denoising, 2020.
[25] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020.
[26] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning contin- uous signed distance functions for shape representation, 2019.
[27] I.. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space, 2019.
[28] R. Liu and C. Vondrick. Humans as light bulbs: 3d human reconstruction from thermal reflec- tion. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12531-12542, June 2023.
[29] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 9298-9300 October 2023.
[30] R. Liu, C. Mao, P. Tendulkar, H. Wang, and C. Vondrick. Landscape learning for neural network inversion. In Proceedings the IEEE/CVF International Conference on Computer October 2023.
Vision (ICCV), pages
[31] T. Zhang, H.-X Yu, R. WU, B. Y. Feng, C. Zheng, N. Snavely, J. Wu, and W. T. Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation, 2024.
[32] T. Xie, Z. Zong, Y. Qiu, X. Li, Y. Feng, Yang, and C. Jiang. Physgaussian: Physics- integrated 3d gaussians for generative dynamics, 2024.
[33] B. Duisterhof, Mandi, Y. Yao, J.-W. Liu, M. Z. Shou, S. Song, and J. Ichnowski. Md- splatting: Learning metric deformation from 4d gaussians in highly deformable scenes, 2023.
[34] C. Finn, Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction, 2016.
[35] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, and S. Levine. Time-contrastive networks: Self-supervised learning from video, 2018.
[36] M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell, and S. Levine. Stochastic variational video prediction. arXiv preprint arXiv:1710.11252. 2017.
[37] A. X. Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video prediction, 2018
10

---

[38] D. Suris, R. Liu, and C. Vondrick. Learning the predictability of the future. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12607-12617, June 2021.
[39] A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, 2020.
[40] Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: universal visual represen- tation for robot manipulation, 2022.
[41] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[42] Y. Seo, D. Hafner, H. Liu, F. Liu, S. James, K. Lee, and P. Abheel. Masked world mod- els for visual control. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference Robol Learning, volume 205 of Proceedings of Machine Learning Re- search, pages 1332-1344. PMLR, 14-18 Dec 2023. URL httph:t///ppp http:///pocceegeeee mlr. press/v205 www.seo222aa..... html.
[43] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In K. Liu, D. Kulic, and J. Ichnowski, edi- tors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 416-426. PMLR, 14-18 Dec 2023. URL https: pproceedings.... mlr wttp::sww.........epeeemmmm radosavovic233 p html.
[44] I. Radosavovic, B. Shi, L. Fu, K. Goldberg, T. Darrell, and J. Malik. Robot learning with sensorimotor pre-training, 2023.
[45] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in- the-wild" human videos, 2021.
[46] A. Escontrela, A. Adeniji, W. Yan, A. Jain, X. B. Peng, K. Goldberg, Y. Lee, D. Hafner, and P. Abbeel. Video prediction models as rewards for reinforcement learning, 2023.
[47] Brooks, B. Peebles, Holmes, W. DePue, Y. Guo, Jing, Schnurr, Tay- lor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh. Video genera- tion models as world simulators. 2024. URL attps:://penaae wttpp/pai.coo//eeeeeeeeeeeeeeeee wwt..//ewer..//////oe/// video-generat intrraiiinmmnnn in-ioonmmmlssss as www.wwwww-wssw:lllloll--------d- sorrrdddsmmmatrrr.....
[48] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, Lorenz, Levi, En- glish, Voleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
[49] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align High-resolution video synthesis with latent diffusion models, 2023.
your latents:
[50] S. Zhang, J. Wang, Y. Zhang, K. Zhao, H. Yuan, Z. Qin, X. Wang, D. Zhao, and J. Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models, 2023.
[51] Ho, W. Chan, C. Saharia, J. Whang, Gao, A. Gritsenko, D. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and Salimans. Imagen video: High definition video generation with diffusion models, 2022.
[52] Z. Yang, Y. Chen, J. Wang, S. Manivasagam, W.-C Ma, J. J. Yang, and R. Urtasun. Unisim: neural closed-loop sensor simulator, 2023.
[53] Y. Du, M. Yang, P. Florence, F. Xia. A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel. J. B. Tenenbaum, L. Kaelbling, A. Zeng. and J. Tompson. Video language planning, 2023.
11

---

[54] A. Ajay, S. Han, Y. S. Li, A. Gupta, T. Jaakkola, J. Tenenbaum, L. Kaelbling A. Srivastava, and P. Agrawal. Compositional foundation models for hierarchical planning, 2023.
[55] K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine 7ero- shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310.10639 2023.
12

---

A Implementation Details
A.1 Model Architecture and Training Schedule
We initialize the canonical 3D Gaussians using 10,000 uniformly sampled point cloud the robot mesh, and train the canonical Gaussians for 2,000 steps using the LI image reconstruction loss. Simultaneously, we train our implicit LRS model using the chamfer distance between canonical and posed robot point clouds obtained from simulation. Finally, we train the canonical gaussians, the LRS model, and our appearance deformation network jointly on the L1 image reconstruction loss until convergence. The overall process takes 15-30 minutes on single NVIDIA 3090 GPU depending on the robot embodiment.
Implicit LRS and appearance deformation modules of our method are modeled using 4-layer 256 hidden dimension lightweight MLPs. We encode coordinates using Fourier features as inputs to the networks.
We keep all canonical 3D Gaussian training hyperparameters consistent with the original 3D Gaus- sians codebase, and use the same training hyperparameters across all robot embodiments.
A.2 Text to Robot Pose with CLIP
In section 3.3.1 as our CLIP model, we use gingface. We minimize the dot product between language and image embeddings that are output by their respective towers.
from Hug-
A.3 Text Action Sequences with Video Model
To obtain the generative video model shown in the paper, we fine-tune SVD-XT, which can gener- ate 24-frame videos from image. To enable language conditioning of this model, replace the OpenCLIP
model image embeddings with text embeddings from the same model. With this modification, we fine-tune this model on 256x256 videos from the OpenEmbodi- ment ASU dataset at 10fps following the implementation of [15] until convergence, which takes 12 hours on A100 GPUs. The initial image conditioning and the prompt for the generated videos in the paper are chosen from set of validation episodes not seen during training.
13


Reviews:
[{"review_id": "Y0xT8zbluJ", "reviewer": "Reviewer_VuJm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents an approach for end-to-end differentiable rendering of robot images conditioned on joint angles and based on robot kinematics information. The method consists of 3 components, namely:  -\tRobot’s forward kinematics is used to get a skeleton of the robot in a 2D image from specific viewpoint -\tA modified version of Linear Blend Skinning (LBS) in combination with Gaussian Splatting is used to fit the position of the Gaussians for a pose-dependent the robot mesh -\tAppearance deformation component is used to adjust the visual and shape information components of the Gaussians for further improvement of the rendering The authors show that their approach outperforms baselines in experiments related to rendering quality and robot pose reconstruction, and show several potential applications when combining the approach with foundation models.", "summary_of_recommendation": "The paper shows a methodology that enables end-to-end optimization of robot control policies from images/video and several interesting potential applications. There are no direct experiments on real robots but the paper demonstrates its applicability on different robotic platforms and due to this as well as due to the scope of the paper I think that hardware experiments are not necessary. Due to the sound methodology, the convincing quantitative and qualitative results and the potential applications I recommend accepting the paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- For the self-model quality experiments, you are using nearest neighbor as a baseline, and you only use 1 000 samples from the training set (if I understand correctly) to find the nearest neighbor. However, your model is trained on a much larger training split from the 10k point clouds / 120k renderings respectively. Wouldn’t it be a fairer comparison if you use the complete training split for the nearest neighbor baseline when you compare to your method, and not only 1 000 samples?\n- In the appendix you refer to LRS (do you mean LBS)?\n- There are typos in lines: 139, 141 and 217", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "915n3Wg43z", "reviewer": "Reviewer_trpR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for generating images of robots given their poses. The renderings are done in a differentiable manner. This enables some interesting downstream tasks to be possible such as optimizing robot pose to match a text prompt using a pre-trained image-text encoder.", "summary_of_recommendation": "This is a well written work that presents and interesting and useful new tool for robot learning.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "When optimizing the joint states, do you find that it is highly dependent on the initial joint states? For example, if the initial poses for Figure 4 are substantially far from the ground truth, will the optimization still find the best pose? \n\nCould you please explain how your approach differs from any existing approaches? Is this perhaps the first approach?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EPKeg68mb8", "reviewer": "Reviewer_FkLZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces \"Differentiable Robot Rendering\" (Dr. Robot), a novel approach for creating a differentiable self-model of robots. It integrates Gaussian Splatting, Implicit Linear Blend Skinning, and Pose-Conditioned Appearance Deformation to enable full differentiability from visual appearance to control parameters. The authors demonstrate applications in text-to-robot pose generation, action sequence generation, and visual motion retargeting, aiming to bridge the gap between visual foundation models and robotic control.", "summary_of_recommendation": "While the paper presents an innovative approach with potential significance for robot learning, the lack of substantial real-world validation and several technical limitations are major concerns. The reliance on simulated environments, limited physical robot testing, and simplified scenarios make it difficult to assess the true impact and applicability of the method. The paper would be significantly strengthened by addressing these issues, providing more comprehensive real-world results, theoretical analysis, and ablation studies. In its current form, I recommend major revisions to address these concerns before consideration for acceptance.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Can you provide results from extensive testing on physical robots across various tasks and environments?\n2. How does the computational complexity of your approach compare to existing methods, and is it feasible for real-time control on typical robotic hardware?\n3. How robust is your method to real-world factors such as varying lighting conditions, occlusions, and sensor noise?\n4. Can you provide a more comprehensive comparison with other state-of-the-art methods in robot control and pose estimation, especially for text-to-robot control and motion retargeting?\n5. How well does your method generalize to more complex, multi-step robotic tasks beyond the simple scenarios demonstrated in the paper?\n6. Can you provide ablation studies to justify your design choices and demonstrate the importance of each component in your framework?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the paper is well-written and presents an innovative approach combining Gaussian Splatting with robot kinematics. They highlight the potential impact on bridging visual foundation models with robotics and the method's versatility across robot types. Quantitative results show significant improvements in rendering quality and pose estimation. However, concerns are raised about limited real-world validation, heavy reliance on simulations, and potential computational complexity. Reviewers suggest more comprehensive comparisons with existing methods, ablation studies, and addressing the sim-to-real gap. While the paper demonstrates interesting applications, more rigorous evaluation of practical robotic scenarios and computational efficiency is recommended to strengthen the work.
Following the rebuttal, reviewers comments were addressed, and reviewer FkLZ Raised their scorer to a weak accept.
Decision: Accept