# Expanding Ground Vehicle Autonomy into Unstructured, Off-Road Environments: Dataset Challenges
## 1 Introduction
#### Related Datasets  
The RUG-D dataset [15] aims to address the challenges of semantic understanding in unstructured outdoor environments, particularly for off-road autonomous navigation. It consists of video sequences captured by an onboard camera mounted on a small unmanned ground robot. Unlike other autonomous driving benchmark datasets, RUG-D intentionally includes a diverse range of terrain types, irregular class boundaries, minimal structured markings, and challenging visual properties commonly found in natural environments (see Figure 1). These factors make it a valuable resource for researchers working on tasks such as semantic segmentation, recognition, and detection. The dataset contains over 700 frames, each meticulously annotated with pixel-wise labels. Researchers can use RUG-D to develop and evaluate cutting-edge techniques for various applications, including humanitarian assistance and disaster relief, agricultural robotics, environmental surveying in hazardous areas, and humanitarian demining. By providing a comprehensive and realistic representation of unstructured outdoor scenes, RUG-D contributes to advancing the field of robotics and computer vision in challenging real-world contexts.  
The RELLIS-3D dataset [16] addresses a critical gap by offering data from an off-road environment. It includes annotations for both LiDAR scans and images. The dataset was collected on the Rellis Campus of Texas A&M University, and comprises 13,556 LiDAR scans and 6235 images. Notably, RELLIS-3D introduces unique challenges, such as class imbalance and environmental topography, which differ from urban environments. Current deep learning models for semantic segmentation primarily rely on large-scale datasets from urban scenes. However, off-road environments pose distinct challenges, including unstructured class boundaries, uneven terrain, and irregular features. Figure 2 shows an example image from the RELLIS-3D dataset along with its corresponding semantically labelled ground truth image. Researchers can utilize RELLIS-3D to develop more sophisticated algorithms and explore novel research directions for enhancing autonomous navigation in off-road settings. By combining LiDAR scans, color images, and other sensor data,  
Figure 1: Sample RUG-D imagery and their corresponding semantic segmentation annotations [15].  
this dataset provides valuable resources for improving semantic awareness in challenging real-world scenarios.  
TartanDrive 2.0 [17], an enhanced version of its predecessor TartanDrive 1.0 [18], expands upon its already impressive collection of off-road terrain data. This updated version boasts seven hours of data captured at speeds reaching up to 15 m per second. Sample imagery is shown in Figure 3. Notably, three new LiDAR sensors have been integrated alongside the original camera: inertial, Global Positioning System (GPS), and proprioceptive sensors. These additional modalities significantly enrich the dataset, enabling researchers to delve into self-supervised learning, multi-modal perception, inverse reinforcement learning, and representation learning. The custom infrastructure provided alongside the dataset empowers end users to customize the data to suit their specific platforms. By releasing both the data and the tools used for its collection, processing, and querying, the creators of TartanDrive 2.0 actively encourage collaborative data aggregation. This resource democratizes access to large-scale datasets, fostering advancements in robotics and autonomous systems.