HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose
  Annotations, Affordances, and Reconstructions
Authors: name: Andrew Guo, name: Bowen Wen, name: Jianhe Yuan, name: Jonathan Tremblay, name: Stephen Tyree, name: Jeffrey Smith, name: Stan Birchfield

We present the HANDAL dataset for category-level object pose estimation and
affordance prediction. Unlike previous datasets, ours is focused on
robotics-ready manipulable objects that are of the proper size and shape for
functional grasping by robot manipulators, such as pliers, utensils, and
screwdrivers. Our annotation process is streamlined, requiring only a single
off-the-shelf camera and semi-automated processing, allowing us to produce
high-quality 3D annotations without crowd-sourcing. The dataset consists of
308k annotated image frames from 2.2k videos of 212 real-world objects in 17
categories. We focus on hardware and kitchen tool objects to facilitate
research in practical scenarios in which a robot manipulator needs to interact
with the environment beyond simple pushing or indiscriminate grasping. We
outline the usefulness of our dataset for 6-DoF category-level pose+scale
estimation and related tasks. We also provide 3D reconstructed meshes of all
objects, and we outline some of the bottlenecks to be addressed for
democratizing the collection of datasets like this one.