DOI: 10.5281/zenodo.5085800
title: CHOC: The CORSMAL Hand-Occluded Containers dataset

description: CORSMAL Hand-Occluded Containers (CHOC) is an image-based dataset for category-level 6D object pose and size estimation, affordance segmentation, object detection, object and arm segmentation, and hand+object reconstruction. The dataset has 138,240 pseudo-realistic composite RGB-D images of hand-held containers on top of 30 real backgrounds (mixed-reality set) and 3,951 RGB-D images selected from the CORSMAL Container Manipulation (CCM) dataset (real set). CHOC-AFF is the subset that focuses on the problem of visual affordance segmentation. CHOC-AFF consists of the RGB images, the object and arm segmentation masks, and the affordance segmentation masks. The images of the mixed-reality set are automatically rendered using Blender, and are split into 129,600 images of handheld containers and 8,640 images of objects without hand. Only one synthetic container is rendered for each image. Images are evenly split among 48 unique synthetic objects from three categories, namely 16 <em>boxes</em>, 16 drinking containers without stem (<em>nonstems</em>) and 16 drinking containers with stems (<em>stems</em>), selected from ShapeNetSem. For each object, 6 realistic grasps were manually annotated using GraspIt!: bottom grasp, natural grasp, and top grasp for the left and right hand. The mixed-reality set provides RGB images, depth images, segmentation masks (hand and object), normalised object coordinates images (only object), object meshes, annotated 6D object poses (orientation and translation in 3D with respect to the camera view), and grasp meshes with their MANO parameters. Each image has a resolution of 640x480 pixels. Background images were acquired using an Intel RealSense D435i depth camera, and include 15 indoor and 15 outdoor scenes. All information necessary to re-render the dataset is provided, namely backgrounds, camera intrinsic parameters, lighting, object models, and hand + forearm meshes, and poses; users can complement the existing data with additional annotations. Note: The mixed-reality set was built on top of previous works for the generation of synthetic and mixed-reality datasets, such as OBMan and NOCS-CAMERA. The images of the real set are selected from 180 representative sequences of the CCM dataset. Each image contains a person holding one of the 15 containers during a manipulation occurring in the video prior to a handover (e.g., picking up an empty container, shaking an empty or filled food box, or pouring a content into a cup or drinking glass). For each object instance, sequences were chosen under four randomly sampled conditions, including background and lighting conditions, scenarios (person sitting, with the object on the table; person sitting and already holding the object; person standing while holding the container and then walking towards the table), and filling amount and type. The same sequence is selected from the three fixed camera views (two side and one frontal view) of the CCM setup (60 sequences for each view). Fifteen sequences exhibit the case of the empty container for all fifteen objects, whereas the other sequences have the person filling the container with either pasta, rice or water at 50% or 90% of the full container capacity. The real set has RGB images, depth images and 6D pose annotations. For each sequence, the 6D poses of the containers are manually annotated every 10 frames if the container is visible in at least two views, resulting in a total of 3,951 annotations. Annotations of the 6D poses for the intermediate frames are also provided by using interpolation.<br> <strong>Contacts</strong><br> For enquiries, questions, or comments, please contact Alessio Xompero. For enquiries, questions, or comments about CHOC-AFF, please contact Tommaso Apicella.<br> <strong>References</strong><br> If you work on Visual Affordance Segmentation and you use the subset CHOC-AFF, please see the related work on ACANet and also cite:<br> <em>Affordance segmentation of hand-occluded containers from exocentric images</em><br> T. Apicella, A. Xompero, E. Ragusa, R. Berta, A. Cavallaro, P. Gastaldo<br> IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2023<br> <strong>Additional resources</strong><br> Webpage of 6D pose estimation using CHOC<br> Toolkit to parse and inspect the dataset, or generate new data<br> <strong>Release notes</strong><br> 2023/09/10<br> - Added object affordance segmentation masks<br> <br> 2023/02/08<br> - Fixed NOCS maps due to a missing rotation during the generation<br> - Fixed annotations to include the missing rotation<br> <br> 2023/01/09<br> - Fixed RGB_070001_80000 (wrong files previously)<br> <br> 2022/12/14<br> - Added a mapping dictionary from grasp-IDs to their corresponding MANO-parameters-IDs to grasp.zip<br> - Added object meshes with the NOCS textures/material in object_models.zip<br> - Fixed folder name in annotations.zip<br> - Updated README file to include these changes and fix a typo in the code block to unzip files