DiffCloud: Real-to-Sim from Point Clouds withDifferentiable Simulation and Rendering of Deformable Objects
Priya Sundaresan1† , Rika Antonova1∗ , and Jeannette Bohg1
I. I NTRODUCTION
We consider the real-to-sim problem of inferring parameters of general-purpose simulators from real observations such that the gap between reality and simulation isreduced [1]–[4]. A common solution is to train an inverse model on data generated with a black-box (nondifferentiable) simulator. The input to such models usuallyconsists of trajectories of a low-dimensional state of thesystem, e.g. position and orientation (pose) of rigid objects
in the scene. The output are parameters, such as mass,friction, and other physical object properties. The state of ahighly deformable object cannot be captured by only its pose,
since the object deforms during motion. Hence, we represent
objects with point clouds obtained from depth cameras.
Recent neural network architectures, such as PointNet++ [5]and MeteorNet [6], are well suited for processing point
clouds. As we will show, they can yield inverse models thatoffer a viable solution to the challenging task of real-to-sim
for deformables from point clouds. However, data collection
and training for these can be computationally demanding.
In this work, we propose an alternative approach that
employs a differentiable simulator to allow adjusting simulation parameters directly via gradient descent without
the need for dataset collection and pre-training. Our approach combines differentiable point cloud rendering anddifferentiable simulation to bring the behavior of simulated,highly deformable objects closer to that of real objects.
We instantiate a scene with a simulated object and create
an end-to-end differentiable pipeline that lets us seamlessly
propagate the gradients from real point clouds to the lowlevel physical simulation parameters. For highly deformableobjects, even small changes in these parameters can have asignificant impact on the behavior of the object. We show that
end-to-end differentiability yields a faster alignment between
simulation and reality, compared to training inverse models
with a black-box (i.e. gradient-free) view of the simulator.
We conduct a set of experiments where a robot manipulates highly deformable real objects, such as cloth andpaper towels. We show that our approach successfully infers
simulation parameters, such as mass and stiffness, making
the behavior of simulated deformables match the real ones.
In simulation experiments, we explore interactions of the1 Department of Computer Science, Stanford University, Stanford, CA
94305, USA {priyasun, rika.antonova, bohg}@stanford.edu† P. Sundaresan was supported by the NSF Graduate Research Fellowship.
∗ Supported by the National Science Foundation grant No.2030859 to theComputing Research Association for the CIFellows Project.
This project was supported in part by a research award from Meta.

Fig. 1. Experimental setup. We execute deformable manipulation trajectories using a Kinova Gen3 arm. We post-process observations recorded fromtwo stereo depth cameras (Intel D435) to generate merged point clouds withthe robot arm masked from view. These observations are fed to our proposed
method D IFF C LOUD for real-to-sim parameter estimation.

deformables with rigid objects: stretching a band on a pole,and hanging a vest onto a rigid pole. Overall, our experiments
show that we can obtain similar or better alignment between
the simulated and the target object, compared to the inverse
model baselines. The major benefit of our approach is thatit obtains such an alignment after on average 10 minutesof direct gradient descent, replacing 2.5 – 5.5 hours of datacollection and training for the inverse models.
II. O UR A PPROACH : D IFF C LOUD
Our objective is to discover the physical simulation parameters (e.g. stiffness, mass, friction) that would cause thebehavior of the simulated deformable objects to match thebehavior of observed real objects. We assume that the initialgeometry of objects and location of the grasp are known. We
start by recording a sequence of point cloud observations of
the scene, where a robot manipulates a deformable object.
Then, we construct a simulated environment in a differentiable simulator that can load meshes of deformable &rigid objects and simulate their interactions. For end-to-end
differentiability we implement a differentiable point cloud
sampler, which allows to propagate loss gradients all the way
to the simulation parameters. Figure 2 shows an overview.
Loss Definition : With low-cost depth sensors, we need aloss that avoids paying attention to noise artifacts in the realpoint cloud. Our insight is that such a loss can be obtainedby using a unidirectional Chamfer distance. This yields aloss that relieves the pressure for the simulated point clouds
to match the noisy parts of the real point clouds:
Xsim
xsim − x real ||22 . (1)dsim→real, P real ) =min ||x
Chamf (Px real ∈P realx sim ∈P sim

While the naive computation of the Chamfer distance canbe expensive, PyTorch3D provides an efficient GPU-based

Fig. 2. Overview of D IFF C LOUD: the proposed method for real-to-sim parameter estimation from point clouds. D IFF C LOUD combines differentiable
point cloud sampling with a differentiable mesh-based simulator to propagate losses computed between simulated (blue) and real (green, with red noise
artifacts) point clouds to the underlying simulation properties. We visualize updating mass and stiffness of a simulated cloth lifted off of a table.

implementation (see Section 3.1 in [7]). With that, we canquickly propagate gradients from the point clouds throughthe mesh representation to optimize the low-level physicalparameters of the simulator. We found that including point
clouds from one or two depth cameras is sufficient to construct a partially occluded point cloud that is still informative
enough for the overall optimization to be successful.
We build upon the differentiable simulator DiffSim [8],which supports mesh-based simulation and contact-handlingof rigid objects and thin-shell deformables (e.g. cloth). In
DiffSim, the simulation state is represented by generalized coordinates q = [q1 T , q2 T , . . . , qn T ]T of all objects in the simulation with corresponding velocities q̇ =[q̇T1 , q̇T2 , . . . , q̇Tn ]T . The generalized coordinates of a rigid
body have qi ∈ R6 for rigid object poses. Deformablesconsist of multiple nodes qi ∈ R3 , denoting node positions.
n is the cumulative total of the number of deformable nodes
and rigid bodies in the scene. DiffSim uses the implicit Euler
method to compute q, q̇ at each time step and performs
collision resolution in localized impact zones. A given meshhas a body frame with the origin set to its center of massat the start of simulation. A mesh vertex p has coordinate
p0 = (px , py , pz )T in the body frame and world coordinate
p = f (q) = [r]p0 + t, where r = (ϕ, θ, ψ)T and t =(tx , ty , tz ) is the 6-DoF pose of the mesh. Propagating
gradients from vertex p to the generalized coordinates q
involves computing the Jacobian ∇f and obtaining the partial
derivatives ∂f (q)/∂q. DiffSim can do sim-to-sim alignment
by comparing current mesh states to target mesh states andpropagating gradients from differences in node positions.
To propagate gradients from real point clouds, we implement differentiable point cloud sampling. A triangularmesh face can be represented by its enclosing vertices
p1 , p2 , p3 . In barycentric coordinates, a random point on
the surface of the face can be generated by sampling 3
random numbers (u, v, w) such that u + v + w ≤ 1 [7]. We
can obtain a random point x inside the triangle as: x =up1 + vp2 + wp3 . To generate an N -point, uniform-density
point cloud from a mesh, we first sample N triangular faces{(pi,1 , pi,2 , pi,3 )}i=1,...,N , weighted proportionally to thearea of each face. For the ith sampled face (pi,1 , pi,2 , pi,3 ),we generate random coefficients (ui , vi , wi ) and computea point x i that lies on the face. Concatenating the points
obtained by applying this procedure to the N sampled facesxi }i=1,...,N . We
and coefficients yields the point cloud P = {xconnect the PyTorch3D [7] implementation of this sampling
procedure to the output of a differentiable simulator. Withxi }i=1,...,N can be propagatedthat, gradients from loss on {xto mesh vertices (pi,1 , pi,2 , pi,3 ) via chain rule using:xi /∂pi,1 = ui ∂xxi /∂pi,2 = vi ∂xxi /∂pi,3 = wi∂x(2)III. E XPERIMENTS
Baseline Inverse Models : As baselines, we use methods
that treat simulators as black-box, i.e. don’t differentiate
w.r.t parameters. These inverse models are implemented as
regression networks that take point cloud sequences as inputs
and predict k simulation parameters. They are trained on
simulated point cloud sequences generated by simulations
with various simulation parameters and synthetic noise.
• M ETEOR N ET : An architecture for learning representationsof 3D point cloud sequences from [6], which we modify
to predict k simulation parameters. We use MeteorNet-cls(Appendix D.2 in [6]) for this regressor.
• P OINT N ET ++: A regressor similar to the above, but using
the multi-scale group architecture from [5] (Appendix B.1)
to extract features from a single point cloud; uses three set
abstraction layers followed by three fully connected layerswith output sizes (512, 256, k).
• MLP: A regressor with a fully connected network that
also operates on a single frame; uses five 2D convolutional layers with output sizes (64, 64, 64, 128, 1024), a
symmetric max pooling layer, two fully connected layerswith output sizes (512, 256), a dropout layer, and a final
fully connected layer with k outputs.
As training data for the regressors we initialize N simulations with uniformly sampled parameters and record theresulting point cloud sequencesand ground truth parameters.
Nxt }Tt=1 , w = [wstiff , wmass ] i=1 .
This yields a dataset D = {xWe use 1500 training and 375 test point cloud sequences withFig. 3. Left: A Kinova robot executes trajectories to lift real cloths froma flat starting state. From these trajectories, D IFF C LOUD accurately infers
stiffness and mass parameters capturing the observed degree of collapsibility
in the real cloths. Right: Across all cloth types, D IFF C LOUD achieves lower
loss on average than all competing baselines on the lift scenario.

Fig. 4. Left: D IFF C LOUD correctly learns to approximate low mass/highstiffness for the shape retaining cloth (1st row) such that three corners lift
off the table mid-fold (2nd row, 2nd column), and high mass/low stiffness
result for the heavy, highly deformable polka dot fabric (3rd row) such thatall ungrasped corners rest on the table mid-fold (4th row, 2nd column).
Right: Compared to data-driven baselines, D IFF C LOUD achieves lower orcomparable loss in 13/15 trajectories across categories. Due to difficulties
perceiving very thin sheets in motion, all methods struggle with 2/15 paper
towel (shape retaining) trajectories, which appear as outliers.

3,500 points per frame across methods. M ETEOR N ET regresx}Ti=1 → w from input point cloud
sor learns a mapping f : {xTx
sequence {x t }t=1 to simulator parameters w . P OINT N ET ++and MLP methods learn a mapping g : xt → w . We use
the same xt as the one used to compute the unidirectional
Chamfer distance in D IFF C LOUD optimization. For eachbaseline, using the target point cloud trajectory as input, wefirst infer the predicted parameters. These serve as input tothe simulator; we then run the simulator and generate point
clouds from the simulated meshes. We compute the Chamferdistance between the point clouds generated by the baselines
and the real point cloud, then compare this against the loss
from running D IFF C LOUD on the real point cloud.
Real Robot Experiments : Figures 3,4 show lift andfold scenarios with 5 different fabrics grouped into three
categories: highly deformable, medium, and shape retaining.
For each fabric, we execute 3 trajectories (15 trajectoriesper scenario), ≈ 2.5 seconds each. During lift and fold thecloth either collapses or maintains its shape depending on
the underlying cloth properties – stiffness and mass, which
we aim to learn. D IFF C LOUD achieves a lower loss than thebaselines across the 3 cloth categories in the lift scenario(Figure 3). The final parameters found by D IFF C LOUD alignwith the qualitative descriptions of the three categories ofcloth types considered (Figure 5). Self-occlusion is more
severe in the fold scenario, yet D IFF C LOUD achieves aloss on par with the baselines, which are (a) trained fromthousands of examples, (b) use data augmentations to beinvariant to varying degrees of self-occlusion, and (c) requiresignificantly more compute time. While baseline regressors
run inference in milliseconds, each baseline takes >2 hours
for dataset generation and additional 40 minutes to multiple
hours per scenario for training. D IFF C LOUD takes on average
10 minutes per trajectory (Figure 5). With significantly lesscomputational footprint, D IFF C LOUD achieves parameter
estimation results on real data that are comparable and insome cases better than baselines.
Further Simulation Experiments : Figure 6 shows evaluation on contact-rich simulated scenarios: hanging one

Fig. 5. Left: We visualize the average compute times across all methods
for performing parameter estimation in the real lift and fold scenarios.
Compared to the baseline inverse models, D IFF C LOUD achieves more thanan order of magnitude speedup, since it eliminates the need to pre-generate adataset and train on it. Right: The optimized D IFF C LOUD parameters foundin the lift scenario correspond to the intuitive physical properties of realcloths, ranging from highly deformable to shape retaining. Each darkened
circle represents the category median across three trajectories per cloth type.

shoulder of a vest onto a pole (vest hang); stretching anelastic band against a pole (band stretch). The massand stiffness of the vest mesh determine the outline of thevest when hung on the pole. The mass and stiffness of theelastic band dictate the extent to which the band travels up
or down along the pole when pulled taut. For evaluation,
we generate a held-out test set of 10 simulated episodes foreach scenario with mass and stiffness parameters sampled
uniformly from [0.1, 10] range. Since synthetic point clouds
are noiseless, it is possible to choose a termination criteria
for D IFF C LOUD based on when the computed loss falls
below a some threshold. We find that loss<5e-4 corresponds
to a well-aligned match, hence, we run D IFF C LOUD until
the loss falls below this threshold or until the number ofoptimization iterations exceeds 50 (same as for lift,fold).
D IFF C LOUD converges to a set of parameters that yield aChamfer distance below the threshold in all cases, while thebaselines only reach sub-threshold alignment in 50-80% ofruns (Figure 6). D IFF C LOUD optimization for a given target
point cloud sequence takes 5 minutes averaged across thevest hang and band stretch scenarios, compared toa combined dataset generation, training, and inference timeon the order of hours for the baselines.

Fig. 6. Two contact-rich, simulated scenarios: band stretch and vesthang. Starting from an initial guess for the parameters (black), D IFF C LOUD
takes gradient steps to update the parameters such that the loss taken
between point cloud observations with optimized parameters (blue) andtarget point clouds (green) is minimized. The optimization terminates once
the loss falls below a threshold of 5e-4, denoting close visual alignment.
D IFF C LOUD optimizes for the mass and stiffness parameters of simulated
deformables to match a highly deformable target elastic band (top row) anda shape retaining target vest (bottom row). Across 10 runs per scenario,
D IFF C LOUD achieves alignment between simulated and target point clouds
in all runs, while baseline regressors yield mixed success (right bar plots).

R EFERENCES[1] A. Prakash, S. Debnath, J.-F. Lafleche, E. Cameracci, S. Birchfield,
M. T. Law et al., “Self-supervised real-to-sim scene generation,” inProceedings of the IEEE/CVF International Conference on Computer
Vision, 2021, pp. 16 044–16 054.
[2] P. Chang and T. Padif, “Sim2real2sim: Bridging the gap between
simulation and real-world in flexible object manipulation,” in 2020
Fourth IEEE International Conference on Robotic Computing (IRC).
IEEE, 2020, pp. 56–62.
[3] J. Zhang, L. Tai, P. Yun, Y. Xiong, M. Liu, J. Boedecker, andW. Burgard, “Vr-goggles for robots: Real-to-sim domain adaptation forvisual control,” IEEE Robotics and Automation Letters, vol. 4, no. 2,
pp. 1148–1155, 2019.
[4] F. Liu, Z. Li, Y. Han, J. Lu, F. Richter, and M. C. Yip, “Real-to-simregistration of deformable soft tissue with position-based dynamics for
surgical robot autonomy,” in 2021 IEEE International Conference on
Robotics and Automation (ICRA). IEEE, 2021, pp. 12 328–12 334.
[5] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchicalfeature learning on point sets in a metric space,” Advances in neural
information processing systems, vol. 30, 2017.
[6] X. Liu, M. Yan, and J. Bohg, “Meteornet: Deep learning on dynamic
3d point cloud sequences,” in ICCV, 2019.
[7] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson,
and G. Gkioxari, “Accelerating 3d deep learning with pytorch3d,” arXiv
preprint arXiv:2007.08501, 2020.
[8] Y.-L. Qiao, J. Liang, V. Koltun, and M. Lin, “Scalable differentiablephysics for learning and control,” in International Conference on
Machine Learning. PMLR, 2020, pp. 7847–7856.