DOI: 10.5061/dryad.ksn02v7hf
title: POLAR-Sim: Augmenting NASA's POLAR dataset for data-driven lunar perception and rover simulation

description: NASA's POLAR (Polar Optical Lunar Analog Reconstruction) dataset
 contains approximately 2,600 pairs of high dynamic range stereo photos
 captured across 12 varied terrain scenes, including areas with sparse or
 dense rock distributions, craters, and rocks of different sizes. The
 purpose of these photos is to spur research and development in robotics,
 AI-based perception, and autonomous navigation. Acknowledging a scarcity
 of lunar photos from around the lunar poles, NASA Ames produced on Earth
 but in controlled conditions, photos that resemble rover operating
 conditions from these regions of the Moon.   This dataset, named
 POLAR-Sim, provides bounding boxes and semantic segmentation information
 for all the photos in NASA's POLAR dataset. This effort results in
 23,000 labels and semantic segmentation information pertaining to rocks
 and shadows of rocks. Furthermore, for each scene, we produced individual
 meshes associated with the ground and the rocks in each scene. This allows
 anyone with a camera model to generate synthetic images associated with
 any of the 12 scenarios of the POLAR dataset. Effectively, one can
 generate as many semantically labeled synthetic images as desired -- from
 different viewpoints in the scene, with different exposure values, for
 different positions of the Sun, with or without the presence of active
 illumination, etc.   The benefit of this work is twofold. Using
 outcomes of the photo annotations, one can train and/or test perception
 algorithms that deal with Moon photos. For meshes of the scenes, one can
 produce as much data as desired to train and test AI algorithms that are
 anticipated to be used in lunar conditions. All the outcomes of this work
 are available in a public repository for unfettered use and distribution.
description: <strong>Photo Bounding Box
 Annotation</strong> To support the
 training of data-driven perception algorithms, we manually labeled
 bounding boxes for all of the rocks and rocks' shadows in the POLAR
 dataset. This effort was motivated by the observation that object
 detection for rocks and shadows plays an important role in autonomous
 navigation -- large rocks can block the rover's path, while medium
 and small rocks can damage the wheels or the chassis. Shadows also help
 estimate the Sun's position, which is vital for navigation planning,
 solar energy harvesting, and sensor orientation. <br>
 Approximately 23,000 rocks and rocks' shadows were
 labeled. Each photo's configuration includes the terrain ID, stereo
 camera position (A: 1.5 m from terrain center at 0 deg, B: 4 m from
 terrain center at 0 deg, or C: 1.5 m from terrain center at 280 deg),
 rover light status (ON or OFF), Sun azimuth angle (none, 30, 180, 270, or
 350 degrees), stereo camera index (Left or Right), and exposure time (32
 to 2048 ms), where "none" means no simulated Sun used. Each
 POLAR photo was taken under a combination of these configuration
 parameters. Note that the labels for the rocks and shadows remained the
 same for several POLAR photos. Specifically, exposure time variations did
 not alter the positions of the rocks and their shadows, so photos with
 exposure times of 32, 64, and 128 ms share the same labels. Other
 similarities include: different rover light statuses have the same rock
 and shadow labels, same camera positions with different Sun azimuths have
 the same rock labels but different shadow labels, and the Left and Right
 camera views have similar rock and shadow label positions. Leveraging
 these observations reduced the burden of labeling. Finally, since in the
 POLAR dataset the exposure time was controlled, at very low exposure time
 the shadow labels were deleted subjectively by the human annotator, since
 it was dark enough for the shadow to be judged as
 "invisible." <br> Annotation was
 performed using <em>labelImg</em>. For each terrain, the first
 Left stereo photo with rover light OFF was manually labeled. Labels were
 reused for photos with different exposure times, with shadow labels
 omitted for very low exposures due to their invisibility. The labels were
 then replicated and adjusted for the Right camera view with similar rock
 and shadow positions, and were replicated for photos with different Sun
 azimuths which only required shadow adjustments. Thereafter, labels were
 replicated and fine-tuned for photos with rover light ON. This process was
 repeated for all stereo camera positions (A, B, and C) and for all 13
 POLAR terrain scenarios. The annotations were saved in YOLO format as TXT
 files.<br><br> 
 <strong>Photo Semantic Segmentation Map
 Annotation</strong> In POLAR-Sim,
 we provide semantic segmentation labels, manually annotating the
 background, ground, rocks, and rock shadows in the photos using
 <em>Roboflow</em>. A similar procedure to the bounding box
 annotation was applied, leveraging repeated segmentation across photos of
 similar configurations. For each terrain, the clearest photo with suitable
 brightness was selected for accurate annotation, refined after using the
 Segment Anything Model (SAM). The resulting segmentation files were then
 copied and fine-tuned for other photos of similar configurations. This
 method ensured consistent and high-quality segmentation across the
 dataset. Segmentation annotations were also saved in YOLO format as TXT
 files, with a converter program provided to output the conventional
 gray-scale map formats. 
 <strong>Mesh Construction of the Ground and
 Rocks</strong> 
 On the other hand, we produced mesh files of the ground and
 rocks for each terrain scenario of the POLAR dataset. Manually locating
 the rocks and generating surface meshes was performed in MATLAB, using the
 point cloud data provided in the POLAR dataset. First, for each terrain
 scenario, two point clouds, respectively scanned from camera positions A
 and C (positions were defined in the 2nd paragraph in the Photo Bounding
 Box Annotation section), were inversely transformed back to the sandbox
 coordinates (where +X corresponds to Sun azimuth 0 deg, +Y to 90 deg, and
 +Z is upward). To enhance the mesh completeness, e.g., mitigating voids
 and occlusions from a single viewpoint, and to better capture the 3D shape
 of each rock, the two point clouds were manually coarse-aligned and
 positioned at the origin. Each rock's point clouds were then further
 locally fine-aligned to minimize occlusions and recover as much of the
 rock's shape as possible. Finally, the annotators manually identified
 the X, Y, and Z coordinate ranges of the rock to form a bounding
 cuboid. <br> Once all the rocks in the
 terrain were located, the point clouds of the rocks and ground were
 separated. These separated point clouds were then converted into meshes
 using the Poisson Surface Reconstruction method in MATLAB, and the results
 were stored as OBJ files. 
 <strong>Simulation Videos</strong>
 A collection of simulation videos is available. All
 videos were made at 0.5x speed. Videos are named in the following format:
 [camera viewpoint]_[Sun ID]_[BRDF]_ [exposure time].mp4. The [camera
 viewpoint] includes two third-person-view cameras from the left and right
 sides of the VIPER, <em>CamBirdViewLeft</em> and
 <em>CamBirdViewRight</em>, four wheel cameras,
 <em>WheelCam_RightFront</em>,
 <em>WheelCam_RightBack</em>,
 <em>WheelCam_LeftFront</em>, and
 <em>WheelCam_LeftBack</em>, and the front-end camera,
 <em>front_end_cam</em>, respectively. The [Sun ID] annotates
 the Sun's directions, where 1, 2, 3, and 4 represent East, Southeast,
 Southwest, and West, respectively. <em>hapke</em> or
 <em>default</em> in the [BRDF] specifies either the Hapke or
 Principled BRDF in Chrono::Sensor. And [exposure time] is set to 0256,
 0512, or 1024 milliseconds.
description: # POLAR-Sim
 [https://doi.org/10.5061/dryad.ksn02v7hf](https://doi.org/10.5061/dryad.ksn02v7hf) A database of bounding box and semantic segmentation labels and terrain meshes for the [POLAR dataset](https://ti.arc.nasa.gov/dataset/IRG_PolarDB/) ## Cover photos of rock indices Pictures in the *CoverPhotoOfIndices.zip* folder show how we index the rocks in each terrain. The indices do not meet the label orders in the bounding box label txt files. The indices meet the rock ID of the mesh files in each terrain. Original pictures come from the POLAR dataset. # Semantic segmentation labels Please check the *SegmentLabels_Terrain[terrain ID].zip* folders. The annotations were done with Roboflow. The semantic segmentation label files in YOLO format are categorized in the terrain ID folders. Each txt file corresponds with one HDR photo of the POLAR dataset. * The label files are named as the following rule: \[terrain ID] \_ [stereo camera position] \_ [rover light on/off] \_ [Sun azimuth] \_ [Left/Right camera of the stereo camera] \_ [exposure time in millisecond].txt * "no" in [Sun azimuth] means that there was no simulated Sun (the spot light was turned off). * For [stereo camera position], A: 1.5 m from terrain center at 0 deg, B: 4 m from terrain center at 0 deg, and C: 1.5 m from terrain center at 280 deg. Details can be referred to Figure 2 on Page 7 and Table 1 on page 8 in the [README](https://ti.arc.nasa.gov/dataset/IRG_PolarDB/PolarDB_Readme.pdf) document of the original POLAR dataset. In the YOLO text files, here are the following classes and their corresponding object: * class 0 is background * class 1 is ground * class 2 is rock * class 3 is rock's shadow To convert a YOLO text file and corresponding POLAR image to a masked bitmap format, you can use the *yolo2bitmap.py* script. ## Bounding box labels Please check the *BboxLabels_Terrain[terrain ID].zip* folders. The bounding box label files in YOLO format are categorized in the terrain ID folders. Each txt file meets with one HDR photo of the POLAR dataset. * The label files are named as the following rule: \[terrain ID] \_ [stereo camera position] \_ [rover light on/off] \_ [Sun azimuth] \_ [Left/Right camera of the stereo camera] \_ [exposure time in millisecond].txt * "no" in [Sun azimuth] means that there was no simulated Sun (the spot light was turned off). * For [stereo camera position], A: 1.5 m from terrain center at 0 deg, B: 4 m from terrain center at 0 deg, and C: 1.5 m from terrain center at 280 deg. Details can be referred to Figure 2 on Page 7 and Table 1 on page 8 in the [README](https://ti.arc.nasa.gov/dataset/IRG_PolarDB/PolarDB_Readme.pdf) document of the original POLAR dataset. * The photos of very low exposure time may not have shadow or rock labels, since the photos are so dark that the rocks and shadows are judged invisible by the annotator. * If the photos have no corresponding label files, it means that the photos have no labels (usually due to nothing visible). In the YOLO text files, here are the following classes and their corresponding object: * class 0 is rock * class 1 is rock's shadow as the same order of the class names in the *classes.txt* in all the *BboxLabels_Terrain[terrain ID]* folders. ## Separated ground mesh and rock meshes of the terrain Please check the *Meshes_Terrain[terrain ID].zip* folders. Meshes of the separated ground and rocks (rock IDs can be referred in the cover photos) of each terrain are built in obj files. * The ground mesh for each terrain, named *terrain[terrain ID] \_ ground.obj*, was generated by removing all the rocks from the terrain. * The mesh files of rocks are named as the following rule: terrain[terrain ID] \_ rock[rock ID].obj * The coordinate directions are defined as follows,\ +X: Sun azimuth 0 deg, +Y: Sun azimuth 90 deg, and +Z: upward of the sandbox * The coordinate order in the obj files is defined as: (X, Y, Z). * Those obj files with file name suffixed by "decimate-005" have only 5% vertices of original object meshes. Those files are used in contact models for collision detection/calculation (such as in PyBullet or Chrono) to mitigate computing loading and facilitate computing speed. * Terrain 4: Rocks 16 and 17 in the cover photo were combined into Rock 16 mesh obj file, since they were indivisible along X, Y, or Z axis. * Users can use the freely-accessible [Blender](https://www.blender.org/download/) software to open the OBJ files, with setting Forward Axis as +Y and Up Axis as +Z when importing the OBJ files in Blender. ## Report Methodology to construct this dataset, several use cases and software/models used to attest the dataset performance, and the corresponding dataset performance results are reported in our [preprint paper](https://arxiv.org/abs/2309.12397). ## Simulation videos *Videos.zip* contains the complete collection of simulation videos. All videos were made at 0.5x speed. Videos are named in the following format: [camera viewpoint] \_ [Sun ID] \_ [BRDF] \_ [exposure time].mp4. The [camera viewpoint] includes two third-person-view cameras from the left and right sides of the VIPER, *CamBirdViewLeft* and *CamBirdViewRight*, four wheel cameras, *WheelCam_RightFront*, *WheelCam_RightBack*, *WheelCam_LeftFront*, and *WheelCam_LeftBack*, and the front-end camera, *front_end_cam*, respectively. The [Sun ID] annotates the Sun's directions, where 1, 2, 3, and 4 represent East, Southeast, Southwest, and West, respectively. *hapke* or *default* in the [BRDF] specifies either the Hapke or Principled BRDF in Chrono::Sensor. And [exposure time] is set to 0256, 0512, or 1024 milliseconds. ## Contributors Advisor: Prof. Dan Negrut Coordinators: [Bo-Hsun Chen](https://github.com/DanielYamChen/) (mesh construction and bounding box labeling), [Thomas Liang](https://github.com/lliangthomas) (semantic segmentation labeling) Bounding box labeling: Bo-Hsun Chen (Terrain 1), Peter Negrut (the other terrains) Terrain digitization: Bo-Hsun Chen (Terrains 1, 4, 11), Thomas Liang (Terrain 9), Peter Negrut (the other terrains) Semantic segmentation labeling: Peter Negrut (Terrains 1, 2, 3, 10, 12, 13), Alice Yang (Terrains 4, 5, 7, 8, 9, 11) We would also like to express our acknowledgements to the high school students Abhinav Annasamudram, Brenden Bushman, Vaidya Srikanth, Li Ying, Shivani Potnuru, and Manushri Muthukumaran for their assistance at the beginning of the semantic segmentation annotation process.