# Expanding Ground Vehicle Autonomy into Unstructured, Off-Road Environments: Dataset Challenges
## 3 Off-Road Autonomy Dataset: A Preview
### Proposed Path Forward for Dataset Curation Supporting Off-Road Autonomy
#### 3.1.1 High-Level, Categorical Class Labels  
We propose broadening the labels to high-level classes rather than very specific class labels that require pixel-perfect labeling. The complexity of obtaining pixel-perfect annotations suggests that the only feasible approach is through simulation strategies for building training datasets. While modern technologies like UE5 have become increasingly impressive for generating photo-realistic datasets, using strictly synthetic data for training AI models has yet to be proven as a viable method. We agree with the literature that emphasizes the value of using both synthetic and real-world imagery for training AI models [19; 20; 21; 22]. Synthetic data can serve as a valuable supplement to real-world datasets. Therefore, we propose a generalized categorical approach to class labels. For instance, paths, regardless of their material composition, should be labeled as the general class label 'path' rather than breaking them down into different types of paths, such as gravel, dirt, grass, etc. To address this, it is important to integrate contextual information or develop multi-stage models that first classify the broader \"path\" category before optionally refining specific sub-classes during post-processing or further model stages. Additionally, while synthetic data plays a crucial role in supplementing real-world data, domain adaptation  
Figure 5: Subset of thermal imagery to be included in our upcoming open-source off-road dataset.  
techniques, such as style transfer or domain randomization, should be applied to bridge the gap between synthetic and real-world data. These methods can help the model generalize better by exposing it to variations and noise typical of off-road environments.  
Another key consideration we acknowledge is the potential impact of generalized labels on model interpretability and decision-making in critical scenarios. In off-road environments, small but significant terrain variations, such as the difference between wet grass and dry dirt, can alter the vehicle's required response. Over-generalizing class labels might reduce the model's ability to make fine-grained distinctions that are crucial for safe navigation. One way to mitigate this issue is to incorporate sensor fusion techniques, combining LiDAR, radar, and camera data to offer richer contextual insights beyond visual labels alone. Multi-sensor data allows models to infer underlying terrain properties like slipperiness or obstacle density, which are not easily captured through image segmentation alone. Additionally, hierarchical labeling systems could be employed, where high-level categories like \"path\" are used during initial training phases, but models can progressively refine predictions based on learned environmental context or specific downstream tasks. This layered approach can ensure that while data annotation remains manageable, the AI still retains the capability to differentiate critical subtleties in off-road terrains when needed. Combining these strategies--sensor fusion, hierarchical labeling, and domain adaptation--creates a more robust and adaptable model, capable of overcoming the inherent challenges of imperfect off-road datasets.  
Figure 6 shows an example semantic segmentation map generated by a model that was trained using the RELLIS-3D dataset for an image from the dataset we are currently developing. As seen in Figure 6, a class label is shown in a light blue color that we suggest should not be differentiated from that predicted path shown in orange. We believe it is imperative to avoid hindering progress in this field due to the immense complexity involved in obtaining pixel-perfect labeling. There must be a level of acceptance that off-road autonomy inherently means training AI models on 'dirty' data, which means semantic labels will be imperfect. To address this and foster growth in this area, we suggest higher-level categorical labels for classes.