# Expanding Ground Vehicle Autonomy into Unstructured, Off-Road Environments: Dataset Challenges
## 3 Off-Road Autonomy Dataset: A Preview
### Proposed Path Forward for Dataset Curation Supporting Off-Road Autonomy
#### 3.1.2 Embracing Imperfect Class Labels  
Considering the challenges in achieving pixel-perfect labels, we propose shifting our focus to training AI models to identify the capabilities of UGVs based on perceived conditions and sensed system-level conditions. For instance, we could train models to recognize tire slippage based on throttle levels and real-time proprioceptive sensor measurements. By making generalizations about the scene, we can enable more sophisticated autonomy behaviors that ensure safe and successful navigation. This could lead to the development of new autonomy stacks that deviate from the conventional paradigm associated with  
Figure 6: Example output (**right**) from a model trained on the RELLIS dataset, applied to an image (**left**) from our custom dataset. Here, it is emphasized the need for a broadening of the labels to high-level classes rather than being differentiated between grass and dirt, for example. Focusing on the transition phase in the image from gravel (labeled with the orange color) to grass (labeled with the light blue color), we question whether this level of granularity is potentially having a net-negative effect on the progress of off-road autonomy. urban/structured environments and adopt a high-level understanding of the scene, with a stronger emphasis on on-board sensing and advanced processing of critical areas. This also is an open opportunity for investigating fuzzy logic-based methodologies [23; 24] for off-road autonomy applications. Figure 7 shows a set of example outputs, corresponding to the images shown in Figure 4, for a model trained for semantic segmentation using the RELLIS dataset. While the model qualitatively appears to do a decent job with its predicted pixel-wise labels, the requirement of differentiating between such nuanced classes leads to predicted labels that complicate an autonomy stack's ability to utilize its information efficiently. Embracing imperfect class labels can actually enhance the robustness of AI models in off-road autonomy by encouraging adaptability and resilience in uncertain environments. In real-world scenarios, perfect annotations are often unrealistic, particularly in unstructured terrains where class boundaries are ambiguous and environmental conditions fluctuate. By training models on imperfect labels, we promote the development of systems that can generalize better to unseen, real-world data. These models become less reliant on pixel-perfect segmentation, and more focused on extracting meaningful, high-level information from noisy or incomplete inputs. Furthermore, imperfect labels mimic the variability and noise that UGVs encounter in off-road environments, allowing the models to learn how to navigate with less-than-perfect data. This can reduce overfitting, making the autonomy stack more flexible and capable of handling novel terrains. Additionally, by accepting imperfect labels, we expedite the data annotation process, significantly reducing the cost and time associated with manually labeling vast amounts of off-road data. Ultimately, embracing imperfection aligns with the realities of off-road autonomy, fostering models that prioritize practicality and robustness over unattainable accuracy in labeling. Using imperfect labels can actually be a significant benefit for off-road autonomy, as it pushes AI models to develop a higher degree of adaptability and robustness. One potential solution is leveraging weak supervision, where models learn to make sense of patterns in data with incomplete or noisy labels. This aligns well with the unpredictable and unstructured nature of off-road environments, as models trained this way are inherently better at generalizing to new and uncertain scenarios.