VIRDO: Visio-tactile Implicit Representations of Deformable Objects
Youngsun Wi1 , Pete Florence2 , Andy Zeng2 and Nima Fazeli1

Abstract— Deformable object manipulation requires computationally efficient representations that are compatible withrobotic sensing modalities. In this paper, we present VIRDO:
an implicit, multi-modal, and continuous representation for
deformable-elastic objects. VIRDO operates directly on visual (point cloud) and tactile (reaction forces) modalities andlearns rich latent embeddings of contact locations and forces
to predict object deformations subject to external contacts.
Here, we demonstrate VIRDOs ability to: i) produce highfidelity cross-modal reconstructions with dense unsupervisedcorrespondences, ii) generalize to unseen contact formations,and iii) state-estimation with partial visio-tactile feedback.

3D scanners commonly found in industry. Contact locations
are also given as a set of points Q which can be given by
an upstream perception algorithm such as [6]–[8]. For thetactile input, net reaction force is given by u ∈ R3 at the wristof the robot which can be measured by common industrialF/T sensors or recovered from joint torques. In this paper,
we derive a continuous and implicit representation of theP, Q , u ) = s) subject to external
deformed object geometry ( f (Pforces and their locations. Here the object geometry is givenby the zero-level set of the implicit function; i.e. s = 0.

I. I NTRODUCTION

II. R EPRESENTING D EFORMABLE O BJECTS USING SDF S

Dexterous manipulation of deformable objects is an important open problem in robotics [1], [2]. These objects
are ubiquitous in our day-to-day lives and play a key rolein many applications including cooking, manufacturing, inhome assistive care, and surgery. Despite their prevalence andimportance, deformable objects have received less attention
than their rigid counterparts owing to their inherent complexities in modeling, perception, and controls [2]–[5]. To
illustrate, the states of rigid bodies with known geometries
can be succinctly represented with 6D pose and velocity.
However, deformable objects have an infinite continuum ofstates and their representation and perception remains anopen problem [2].
In this paper, we present VIRDO – an implicit, dense,cross modal, and continuous architecture that addresses thesefundamental representation and perception challenges for theclass of elastically deformable objects. The central feature of
our method is learning deformation fields informed by crossmodal visual and tactile cues of external contacts. We further
contribute a dataset of elastically deformable objects withboundary conditions used to evaluate. This paper focuses ondense geometric representations because they can facilitate
downstream tasks such as state-estimation from partial viewsand estimating dense correspondences, as we demonstrate, as
well as bootstrapping keypoint/affordance learning.
Problem Statement & Assumptions : Our goal is toderive a computationally efficient and generative model that:
1) predicts object deformations subject to external forces;and 2) is compatible with common robotic sensors. We
assume the object geometry is described by its point cloud:an unordered set P := {pp ∈ R3 : SDF(pp) = 0} where SDF
denotes the signed distance w.r.t. the surface of the object.
Point clouds are obtained from commodity depth sensors or

At a high-level, VIRDO decomposes object representations into a nominal shape representation and a point-wise
deformation field. Here, we choose signed distance fields
as our underlying representation and discuss this choice inSec. IV. The nominal shape representation decodes latentshape embeddings α ∈ Rl into continuous signed-distancefields, the zero-level set of which is the undeformed object
geometry – similar to the architectures proposed in [9]–[11].
The point-wise deformation field is produced using a summary of all boundary conditions (contact locations, reaction
force, and fixed constraints) leveraging a permutation invariant set operator. The structure is fully differentiable and canbe learned end-to-end. In the following, we discuss eachcomponent in more detail.

1 Youngsun Wi and Nima Fazeli are with the Robotics Institute at the
University of Michigan, MI, USA <yswi,nfz>@umich.edu
2 Pete Florence and Andy Zeng with Robotics at Google, Mountain View,
CA, USA <peteflorence,andyzeng>@google.com

A. Nominal Shape Representation
The nominal shape of an object is the geometry it takes inthe absence of external contact forces. The nominal geometry
Ψo (αα )) = s as ais produced by the object module as O (xx|Ψsigned distance field, where x = (x, y, z) is a query point, ands is the signed-distance. The purpose of the object code (α)
is to allow VIRDO to represent multiple objects. Here, weα ) to decode the object code into
use a hyper-network Ψ o (αobject module’s weights and biases θ o , similar to [12].
We pre-train VIRDO on nominal shapes before training
on deformations. During this stage, we initialize the object
codes as α ∼ N(0, 0.12 ) and update them with hyper-networkΨ o . The loss function for pretraining nominal shapes is:Lnominal = Lsd f + λ2 Llatent + λ3 Lhyper ,where Lsd f is defined as:
N Lsd f = ∑i=1

∑ |clamp(OOi (xx), δ ) − clamp(s∗ , δ )|Ωx̄x∈ΩOi (xx), n ∗ ⟩) .
+ λ ∑ (1 − ⟨∇OΩ0x̄x∈Ω(1)

Fig. 1: Representation Architecture: The left panel depicts how visual data in the form of point clouds and tactile in the form of reaction forces may
be collected in practical robotic settings. The middle panel depicts the network and how this information is processed to predict the implicit surface
representation encoded as a signed-distance function. Finally, the right panel depicts the reconstruction of the estimated true surface given the external
contacts and reaction force.

For convenience, we introduce a shorthand O i (xx) =α i )) to denote the signed-distance field correspondO (xx|Ψo (αing to the ith object code. Also, Ω is the 3D querying space,Ω 0 is the zero-level surfaces in the querying space, s ∗ is theground truth signed-distance, n ∗ is the ground truth normal,and ∇ denotes the spatial gradient. δ is for clamping offsurface signed-distances to concentrate network capacity on
details near the surface, as demonstrated in [9]. For latent
space regularization, we impose a Gaussian prior on theα ) = ∑Ni=1 ∥ 1l α i ∥2 and weights of theobject code Llatent (αnetwork Lhyper (θθ o ) = ∑Ni=1 l1o ∥θθ io ∥2 where lo is the lengthof θ o . After the pre-training, we fix the object codes asconstant vectors, but continue updating the object moduleand the hyper-network with low learning rate(∼ 1e-8) foradditional refinement.

This results in D predicting different deformation fields
for different objects despite similar contact locations andreaction force measurements. This is desirable because
objects may be geometrically similar but deform differently
due to varying material properties. We will demonstrateexamples of this in Sec. III-B.
Ψd (zz, α ) , we canUsing the shorthand D Ψ d (xx) = D x |Ψrelate the signed-distance fields of the deformed and nominal object via s = SDF(xx) = O Φ o x + D Ψ d (xx) . We notethat the deformed
 point cloud p ∈ P satisfies SDF(pp) =O Φ o p + D Ψ d (pp) = 0. To learn this mapping, we solve theP), whereoptimization problem argmin fc (PθdP) =fc (PB. Deformed Object Representation
The Force Module F is an encoder that summarizes theQ, u ) into a force code
contact locations and reaction force (QQ, u ). We assume that the contact location set Q isz = F (QQ ⊂ P ) andgiven as a subset of the nominal point cloud (Qthe reaction force u ∈ R3 is directly measured at the robot’swrist. Point clouds, including the contact location set Q ,are unordered and variable in length. Our contact location
encoder utilizes the PointNet architecture [13].
We define the deformation field as a 3D vector field thatpushes a deformed object back to its original (nominal)
shape. VIRDO recovers the signed-distance field of thenominal shape by adding the deformation field to the SDF ofthe deformed shape. We highlight that VIRDO has learned
to focus the deformation field around the boundary of theobject with magnitude reflecting the amount of deformation.
VIRDO represents
thedeformationfieldas
Ψd (zz, α ) , where the deformation module D shares
D x |Ψ
the same structure as the object module O with parameters
θ d predicted by the hyper-network Ψ d . We highlight thatθ d is conditioned on the latent code pair (zz, α ) ∈ Rl+m tocapture the underlying object-specific deformation behavior.

∗

P + D Ψ d (PP), P̄P ) + λcCD(P1DΨ d (pp)∥2∥D∑P p ∈PPP̄(2)∗

P is the true nominal point cloud of the same
where P̄
length as P , λc is a weighting for the minimal correction
prior similar to [14] and CD is the Chamfer Distance measure
between two point clouds.
To learn the full model, we train the deformation, force,and pre-trained object modules end-to-end using the loss
function:
Lde f ormed = fc ({xx|xx ∈ Ω 0 }) + λ1 Lsd f + λ3 Lhyper + λ4 Llatent .
(3)
The first term solves the optimization in Eq. 2 with onsurface points. The second loss term Lsd f couples the deformation field to the signed-distance field of the object module:M Lsd f = ∑i=1

∑ |clamp(OOΦo x + D Ψd (xx), δ − clamp(s∗ , δ )|Ωx ∈Ω

Oi (xx), n ∗ ⟩)+ ∑ λn (1 − ⟨∇O(4)Ω0x ∈Ω

where M is the total number of deformed objects. TheLhyper (xx) and Llatent (xx) are the Gaussian prior on the latentcode and the network parameters. Since we are updating two

Nominal Nominal(G.T.) (Recons.)

Deformed (Recons.)

Nominal(G.T.)

Nominal(Recons.)

Deformed (Recons.)

Fig. 2: Reconstruction Results: Reconstructions of multiple nominal shapes and their deformations, learned simultaneously by VIRDO. Marching Cubealgorithm is used for the reconstruction, where we highlighted the contact location as purple region.

hyper-networks, Lhyper is the weighted sum of Ψ o and Ψ dand the Llatent is ∑Ni=1 ∥ 1l z ∥2 .
III. E XPERIMENTS AND R ESULTS
A. Data Preparation
In total, we generated 6 objects each with 24 uniqueboundary conditions using MATLAB PDE toolbox. The3D meshes were collected from open-sourced 3D model
repository. For the training, we normalized the point cloudwith the geometric center at [0, 0, 0].
B. Representing Known Deformable Shapes
The average reconstruction accuracy is 0.3474x103 in
Chamfer Distance(CD). CD is measured between reconstructions and query points unseen during training in average,
where we utilized Marching Cube algorithm [15] for thereconstruction. We emphasize that only one neural networkmodel was used for the entire data-set.
C. Deformation Field Inference
We test the model’s ability to infer a deformation fieldgiven reaction force, partial pointcloud, and object code,
seen from the training. Here, we infer the contact feature
to estimate deformation. First, we randomly initialize thecontact feature from N(0, 0.012 ). Then, we update the feature
with an L1 loss which only consumes a partial zero-levelO(xx), δ )|. The loss encourages
set: Lin f er = ∑x ∈ΩΩo |clamp(OVIRDO to update the contact feature by minimizing themismatch between the initial guess and the partial observation. Fig. 3 is a partial pointcloud where the handle and thetip are occluded, rendered in simulation with a single pinhole
camera. At epoch 0, the model already makes deformation
field fairly close to the ground truth. This shows VIRDO’s
ability to perform state estimation when the vision is missing.
As the gradient descent progresses on the contact feature, the0[s]0.55[s]Partial Pointcloud[0.2421, 0.0453, -0.0538]
Reaction Force

Object Code0 epoch

60 epochFig. 3: Inference: Reconstructions with inferred deformation field (cyan),ground truth deformed object (magenta)

estimated deformation converges towards the ground truth.
We note that only the on-surface points are used for thisexperiment, since an RGBD camera would feasibly only giveon-surface points in real-world experiments; however, it isalso possible boost the inference performance by collecting
off-surface samples along the camera ray and augment thepartial observation.
D. Generalization and Code Interpolation/Extrapolation
In this section, we evaluate VIRDO’s ability to generalize
to unseen contact formations. This functionality is important
for robotic applications given the wide variety of contact
interactions. To this end, we use the model from Sec. III-B(trained on 144 deformations) and evaluate the reconstruction
accuracy of 6 unseen contact formations for the object

[2]

[3]

[4]
Fig. 4: a) Generalization: Example of shape estimation given unseen
contact formation during training. b) Latent Code Interpolation andExtrapolation: Two trained force codes z l and z r are interpolated andextrapolated evenly as shown where 1-5 indicate tested force code andcorresponding reconstruction result.

depicted in Fig.4a). To better understand the generalization
properties of VIRDO, we interpolate and extrapolate inlatent force code space. This task evaluates the continuity
and semantic meaning of the latent space; e.g. whether
the reconstructions are similar to direct inter/extrapolationof two deformations. Given an object, we pick z l and z r
among the successfully trained force codes; z l bears themaximum deflection to the left (or -x direction) and z r generates moderate deflections in a random direction. We thenlinearly interpolate and extrapolate the 32 dimensional force
code. Fig.4b) shows the resulting reconstructions. We notethe interpolations and extrapolations are smooth, even, andcontinuous. This suggests a well-formed latent force spaceand explains the effectiveness of the model in generalizing
to unseen contact formations.
We note that the reconstruction for extrapolation (1) isomitted from the object on the right in Fig. 4. This isbecause of a failure case resulting in a noisy and poor quality
reconstruction. We found that the extrapolation results can beinaccurate if it exceeds the bound of maximum deflections
seen in the training data.

[5]

[6]

[7]

[8]

[9]

[10]

IV. D ISCUSSION & L IMITATIONS
The fundamental principle driving VIRDO is the ability
to learn deformation fields informed by visio-tactile sensing.
VIRDO is the first learned implicit method to integrate tactile
and visual feedback while modeling object deformations
subject to external contacts. The representation has arbitrary
resolution and is cheap to evaluate for point-wise sampling.
Additionally, the latent code is well-behaved and can be usedfor inference.

[11]

[12]

[13]
V. ACKNOWLEDGEMENT
This research is partly supported by Robotics at Google.
R EFERENCES[1]

A. Billard and D. Kragic, “Trends and challenges
in robot manipulation,” Science, vol. 364, no. 6446,
2019.

[14]

H. Yin, A. Varava, and D. Kragic, “Modeling, learning, perception, and control methods for deformableobject manipulation,” Science Robotics, vol. 6, no. 54,
2021.
J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, andY. Mezouar, “Robotic manipulation and sensing ofdeformable objects in domestic and industrial applications: A survey,” The International Journal of Robotics
Research, vol. 37, no. 7, pp. 688–716, 2018.
V. E. Arriola-Rios, P. Guler, F. Ficuciello, D. Kragic,
B. Siciliano, and J. L. Wyatt, “Modeling of deformableobjects for robotic manipulation: A tutorial and review,” Frontiers in Robotics and AI, vol. 7, p. 82,
2020.
Y. C. Hou, K. S. M. Sahari, and D. N. T. How,“A review on modeling of flexible deformable object for dexterous robotic manipulation,” International
Journal of Advanced Robotic Systems, vol. 16, no. 3,
p. 1 729 881 419 848 894, 2019.
T. Hermans, F. Li, J. M. Rehg, and A. F. Bobick,“Learning contact locations for pushing and orienting
unknown objects,” in 2013 13th IEEE-RAS international conference on humanoid robots (humanoids),IEEE, 2013, pp. 435–442.
M. Sharma and O. Kroemer, “Relational learning forskill preconditions,” arXiv preprint arXiv:2012.01693,
2020.
Q. V. Le, D. Kamm, A. F. Kara, and A. Y. Ng, “Learning to grasp objects with multiple contact points,” in2010 IEEE International Conference on Robotics andAutomation, IEEE, 2010, pp. 5062–5069.
J. J. Park, P. Florence, J. Straub, R. Newcombe, andS. Lovegrove, “Deepsdf: Learning continuous signed
distance functions for shape representation,” in Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, 2019, pp. 165–174.
V. Sitzmann, J. Martel, A. Bergman, D. Lindell, andG. Wetzstein, “Implicit neural representations withperiodic activation functions,” Advances in Neural
Information Processing Systems, vol. 33, 2020.
M. Tancik, P. P. Srinivasan, B. Mildenhall, et al.,“Fourier features let networks learn high frequency
functions in low dimensional domains,” arXiv preprint
arXiv:2006.10739, 2020.
Y. Deng, J. Yang, and X. Tong, “Deformed implicit
field: Modeling 3d shapes with learned dense correspondence,” arXiv preprint arXiv:2011.13650, 2020.
C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet:
Deep learning on point sets for 3d classification andsegmentation,” in Proceedings of the IEEE conferenceon computer vision and pattern recognition, 2017,
pp. 652–660.
Y. Deng, J. Yang, and X. Tong, “Deformed implicit
field: Modeling 3d shapes with learned dense correspondence,” CoRR, vol. abs/2011.13650, 2020. arXiv:
2011 . 13650. [Online]. Available: https : / /arxiv.org/abs/2011.13650.

[15]W. E. Lorensen and H. E. Cline, “Marching cubes:
A high resolution 3d surface construction algorithm,”
ACM siggraph computer graphics, vol. 21, no. 4,
pp. 163–169, 1987.