DOI: 10.22002/d1.1332
title: JPL Mars Yard Database

description: JPL Mars Yard Database is built to understand terrain types from various sensors, such as RGB and IR. It has 2 datasets;

1. Semantic Dataset 1: Understanding Terrain Types from RGB and IR,
2. Virtual Sensor Dataset 1: Deriving RGB-to-IR mapping models.

Both datasets were collected at JPL Mars Yard on Nov. 17th 2017, with a RGB camera (FLIR Grasshopper 5M) and a thermal camera (FLIR AX65). We collected images every 1 hour, from 10 am till 5 pm (sunset), totally 8 times. Total number of images at each time is about 52 images, by changing the position of the cameras. The dataset includes challenging images such as shadows, reflection due to the Sun, and direct sunlight into the cameras.
description: JPL MARS YARD DATABASE, SEMANTIC DATASET 1
Semantic Dataset 1 contains RGB, IR, and annotation images. We manually annotated all images into 7 categories, unlabeled, sand, soil, rocks, bedrock, rocky terrain, and ballast. In the annotation images, pink, brown, green, light blue, purple, and red correspond to sand, soil, rocks, bedrock, rocky terrain, and ballast, respectively.

The dataset was first introduced in the MIPR 2019 paper, TU-Net and TDeepLab: Deep Learning-based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery[1].

The lists of training, validation, and test dataset are provided in the section below. In the paper [1], we evaluated the proposed approach with 3 different settings:

(Exp. 1) train, evaluate, and test with the dataset at 17:00, which has no influence of the Sun,
(Exp. 2) train, evaluate, and test with all dataset from 10:00 to 17:00,
(Exp. 3) train and evaluate with dataset from 14:10 to 17:00, and two tests: (i) test with dataset from 10:00 to 13:00 and (ii) dataset from 14:00 to 17:00.

JPL MARS YARD DATABASE, VIRTUAL SENSOR DATASET 1
Virtual Sensor Dataset 1 contains RGB and IR images. The lists of training, validation, and test dataset are provided below.
The dataset was first introduced in the PBVS 2019 paper, MU-Net: Deep Learning-based Thermal IR Image Estimation from RGB Image[2].

CITATION
If you make use of the JPL Mars Yard Database, Semantic Dataset 1 in any form, please do cite the following paper:
[1] Y. Iwashita, Kazuto Nakashima, Adrian Stoica, Ryo Kurazume, "TU-Net and TDeepLab: Deep Learning-based Terrain Classification Robust to Illumination Changes, Combining Visible and Thermal Imagery", IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR 2019), San Jose, California, USA, 2019.3.28-30, 2019.

If you make use of the JPL Mars Yard Database, Virtual Sensor Dataset 1 in any form, please do cite the following paper:
[2] Y. Iwashita, Kazuto Nakashima, Sir Rafol, Adrian Stoica, Ryo Kurazume, "MU-Net: Deep Learning-based Thermal IR Image Estimation from RGB Image", IEEE Workshop on Perception Beyond the Visible Spectrum (PBVS), Long Beach, CA, USA, 2019.6.16, 2019.

TIPS ON IMPLEMENTATION
At each pixel in the annotation images, it has numeric number which corresponds terrain type.

0: __unlabeled__
1: sand
2: soil
3: ballast
4: rock
5: bedrock
6: rocky terrain

It is encoded and can be read by python as follows:

encoded = np.array(Image.open(path_to_label))
label = np.bitwise_or(np.bitwise_or(
encoded[:, :, 0].astype(np.uint32),
encoded[:, :, 1].astype(np.uint32) << 8),
encoded[:, :, 2].astype(np.uint32) << 16)