MACS: Mass Conditioned 3D Hand and Object Motion Synthesis
Authors: name: Soshi Shimada; name: Franziska Mueller; name: Jan Bednarik; name: Bardia Doosti; name: Bernd Bickel; name: Danhang Tang; name: Vladislav Golyanik; name: Jonathan Taylor; name: Christian Theobalt; name: Thabo Beeler
Published: 2023-12-22 18:59:54

Abstract: The physical properties of an object, such as mass, significantly affect how
we manipulate it with our hands. Surprisingly, this aspect has so far been
neglected in prior work on 3D motion synthesis. To improve the naturalness of
the synthesized 3D hand object motions, this work proposes MACS the first MAss
Conditioned 3D hand and object motion Synthesis approach. Our approach is based
on cascaded diffusion models and generates interactions that plausibly adjust
based on the object mass and interaction type. MACS also accepts a manually
drawn 3D object trajectory as input and synthesizes the natural 3D hand motions
conditioned by the object mass. This flexibility enables MACS to be used for
various downstream applications, such as generating synthetic training data for
ML tasks, fast animation of hands for graphics workflows, and generating
character interactions for computer games. We show experimentally that a
small-scale dataset is sufficient for MACS to reasonably generalize across
interpolated and extrapolated object masses unseen during the training.
Furthermore, MACS shows moderate generalization to unseen objects, thanks to
the mass-conditioned contact labels generated by our surface contact synthesis
model ConNet. Our comprehensive user study confirms that the synthesized 3D
hand-object interactions are highly plausible and realistic.