ARC2017 RGB-D Dataset for Object Detection and Segmentation
Ryosuke Araki1 , Takayoshi Yamashita2 and Hironobu Fujiyoshi3
I. ARC2017 RGB-D DATASETIn July 2017, Amazon Robotics LLC held a competition
called the Amazon Robotics Challenge (ARC) in Nagoya
City, Japan, to compare different logistics automation technologies. Contestants had to complete two tasks — a Stowtask, where robots were required to stow 40 different typesof items from a tote into a storage system, and a Pick task,
where they had to pick items from the storage system andplace them in the cardboard boxes. In the final round, these
two tasks had to be performed consecutively. The items
included various rigid, non-rigid, and translucent objects.
We competed as part of the Team MC2 and created ourown dataset for the ARC2017 tasks. This dataset comprises
images of the 40 items used in the ARC 2017 competition
that were in a red tote. For every scene, the dataset includes
an RGB image, a depth map image, and correctly labeledbounding-box and segmentation data. Image samples areshown in Figs. 1 (a) and (b). This dataset is available onthe website of our research group 4 .
A. RGB Images and Depth Images
The dataset includes 1280 × 960 pixel RGB images anddepth images corresponding to the same scenes. Using theset of 40 ARC2017 items and 20 APC (Amazon Picking
Challenge) 2015 items, we created 1,510 scene images,
as shown in TABLE I. Because the ARC2017 required
recognition of the unknown-class objects, we prepared theAPC2015 items instead.
All images in the dataset were captured using MELFA-3D
vision sensor made by Mitsubishi Electric Corporation. It
incorporates a miniature projector and uses coded structured
light to obtain a depth point cloud.
B. Bounding Boxes
The RGB images are annotated with bounding box information. The annotation files are provided separately foreach image and consist of text data indicating the box
coordinates and the items’ IDs. The annotations are only
found in the regions of the image that can be confirmed
when the annotator is looking at the image, so in the cases
1 Ryosuke Araki is with the Department of Computer Science,
Graduate School of Engineering, Chubu University, Aichi, Japanryorsk@mprg.cs.chubu.ac.jp
2 Takayoshi Yamasita is with Faculty of Computer Science,
CollegeofEngineering,
Chubu
University,Aichi,Japanyamashita@cs.chubu.ac.jp
3 Hironobu Fujiyoshi is with Faculty of Robotics Science andTechnology, College of Engineering, Chubu University, Aichi, Japanhf@cs.chubu.ac.jp
4 http://mprg.jp/en/research/arc_dataset_2017_e

Fig. 1.

Dataset contents for a single scene and examples of 3D models
TABLE IS CENE DETAILS

Train
Test
Total

ARC’17
Known-ARC’17
Unknown-APC’15

Single item
410
410

Multiple items
800
200
100
1,100

Total
1,210
200
100
1,510

where occlusion occurs, the boxes only surround the visible
parts, as shown in Fig. 1 (c).
C. Segmentation Labeled Image
The dataset includes RGB labeled images colored on a perpixel basis in each item region. A different color is used foreach item, and the background is painted black, as shownin Fig. 1 (d). Since the items are painted without antialiasing, they can be used for training and testing of semantic
segmentation.
D. 3D Models
We created 3D models of all 40 items. Fig. 1 (e) showssome examples of the 3D models. These models were createdusing a 3D MFP machine made by Ortery Technologies, Inc.
The models were saved in OBJ file format together withmaterial data in MTL format and texture data in JPG format.
By applying scale information to these models, it is possible
to test recognition and gripping of items in a simulator.
II. C ONCLUSION
We have released the ARC2017 dataset and will show thebenchmark results of the state of the art for object detection
and segmentation by deep learning-based approach such as
Faster-RCNN, Single Shot Multibox Detector, and SegNet.