<figure><img src="image_1.png" /></figure>

HAL
open science

# Improving Off-Road LiDAR Semantic Segmentation with Spatial Context and Auxiliary Tasks

Abhay Dayal Mathur, Alexandre Chapoutot, David Filliat

<figure><img src="image_2.png" /></figure>

## ► To cite this version:

Abhay Dayal Mathur, Alexandre Chapoutot, David Filliat. Improving Off-Road LiDAR Semantic Segmentation with Spatial Context and Auxiliary Tasks. European Robotic Forum, Mar 2025, Stuttgart (Germany), Germany. pp.300-306, 10.1007/978-3-031-89471-8_46. hal-05111999

HAL Id: hal-05111999

https://hal.science/hal-05111999v1

Submitted on 13 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Improving Off-Road LiDAR Semantic Segmentation
with Spatial Context and Auxiliary Tasks

Abhay Dayal Mathur¹,², Alexandre Chapoutot², and David Filliat²

¹ Institut Polytechnique de Paris, Palaiseau, France
abhay.mathur@ip-paris.fr

² U2IS, ENSTA Paris, Institut Polytechnique de Paris, Palaiseau, France
{alexandre.chapoutot, david.filliat}@ensta-paris.fr

**Abstract.** Autonomous navigation for unmanned ground vehicles (UGVs) in complex environments requires robust perception for reliable traversability estimation and path planning. Traditional geometric methods often fail in unstructured terrains, necessitating robust scene understanding. Current 3D semantic segmentation methods are mostly directed towards structured environments and are not generalizable for off-road domains. To address this, we propose modifications to LiDAR semantic segmentation by incorporating spatial context and adding an auxiliary task of learning point-wise true height to capture robust features. Experiments on the Rellis-3D dataset demonstrate that the proposed approach outperforms state-of-the-art methods in segmentation accuracy and adaptability, offering a scalable solution for UGV perception in diverse outdoor environments.

**Keywords:** Lidar semantic segmentation

# 1 Introduction

Autonomous navigation for unmanned ground vehicles (UGVs) in complex and dynamic outdoor environments presents numerous challenges. These vehicles must navigate varying terrains, avoid obstacles in real-time, and maintain stability in diverse conditions. Traditional approaches to perception and navigation rely heavily on geometric representations of the environment, often assuming structured settings and smooth terrains. However, these methods fall short in unstructured environments, where varying slopes, dense vegetation, and unpredictable obstacles can significantly degrade the effectiveness of geometric-based solutions.

Recent advances in deep learning and sensor fusion techniques have shown promise in improving the perception capabilities of UGVs by leveraging rich sensor data like LiDAR point clouds. Specifically, LiDAR-based semantic segmentation can provide a detailed understanding of the surrounding terrain, identifying traversable areas and potential obstacles. Despite the progress, existing segmentation methods often face challenges in domain generalization, especially when deployed in unseen environments with substantial domain shifts, such as changes in terrain, weather, or sensor configurations. This paper proposes an auxiliary learning task to enhance the generalizability of LiDAR-based semantic segmentation in unstructured environments. By learning point-wise true height information as an auxiliary task, our approach aims to encourage the
model to capture domain-agnostic features. We hypothesize that incorporating height estimation improves the model's robustness to unseen domains, allowing it to better generalize across different terrains without requiring prior information about the target environment. Our experiments show that the proposed method significantly improves segmentation accuracy and generalizability on challenging off-road datasets.

# 2 Related work

## 2.1 LiDAR Semantic Segmentation

Several methods have been proposed to address the task of LiDAR Semantic Segmentation (LSS), mostly dealing with structured outdoor environments, such as autonomous driving. Methods like FRNet [1] and SalsaNeXt [2] use the projection of point clouds onto range images, and restore the contextual information of range image pixels using corresponding frustum LiDAR points, or boost performance on sparse scans through residual convolution blocks. SqueezeSegV3 [3] improves upon its predecessors by using context aggregation modules and a new loss function to handle the noise and occlusions in LiDAR data. PointNet++ [4] builds on the original PointNet architecture by using a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set, capturing local structures at different scales. Voxel-based methods convert the point cloud data into a voxel grid and apply 3D convolutions. These methods include architectures like MinkowskiNet [5], which uses sparse convolutions to process voxelized point clouds efficiently.

**LSS for Outdoor Navigation.** 3D semantic segmentation for unstructured environments, however, remains relatively under-explored, and very few publicly available datasets can be used for the task. Recent methods [6,7] use semantic segmentation as an upstream task for estimating traversability from LiDAR scans; by grouping the semantic classes into ‘traversable’ and ‘not-trable’.

## 2.2 Domain Generalization for LiDAR Semantic Segmentation

Domain Generalization for LiDAR Semantic Segmentation (DG-LSS) has lately gathered interest in literature due to its potential to improve model robustness across varied and unseen environments, especially in exploratory navigation where prior knowledge of the environment and ambient conditions is rarely available.

Most efforts in this direction aim at encouraging models to learn domain-agnostic representations through auxiliary tasks [8,9], focusing on developing domain robust representations through leveraging feature density in bird-eye-view (BEV) projections, or through training parallel decoders on surface reconstruction. These methods, however, are specific to structured environments and work under the assumption of either receiving a particular structure in the LiDAR scans [8], or of the surroundings majorly consisting of surfaces [9].

Further, most current methods assume some knowledge of the target domain - either intrinsics for the target sensors [10], a small dataset collected from the target domain to fine-tune the model with, or knowledge of conditions in the target domain to construct bridge domains through simulation [11]. These assumptions rarely hold for exploration in the wild, necessitating robust methods that can generalize over unseen domains.
<figure><img src="image_4.png" /><figcaption>Fig. 1: Overview of the proposed LiDAR semantic segmentation training pipeline. The encoder extracts latent features from the point cloud *P*, which are processed by two parallel decoders: a semantic decoder for segmentation $\hat{y}$ and an auxiliary decoder for ground-relative height prediction $\hat{y}_{aux}$. The combined pipeline is supervised through a composite loss. For the context module, parameters *k*, *d*, and *BN* signify kernel size, dilations and batch-normalization respectively.</figcaption></figure>

# 3 Method

Our method focuses on learning robust representations for LiDAR semantic segmentation through an auxiliary task of learning point-wise true height (de-skewed relative to sensor vibrations) to encourage the model to learn domain-agnostic representations. To this end, we train a decoder on the height-prediction task parallel to the decoder for semantic segmentation directing the encoder to extract geometric features that generalize across domains, reducing reliance on domain-specific information and improving the model's ability to handle diverse environments. An overview of the training pipeline for our method can be found in Fig. 1. Additionally, we incorporate some architectural modification to the Minkowski U-Nets backbone [5] to better take spatial context into account.

## 3.1 Spatial Context

We encourage the encoder model to pay greater attention to the positional information from the point cloud in two ways. Firstly, following our empirical results with using the sparse tensor representation of point clouds in Minkowski Engine [5], we duplicate the coordinates of each point in the point cloud in its feature tensor. Our pilot studies showed that this simple modification significantly improves the performance of the model, and obviates the need for hand-engineered features obtained from storing scan statistics in a global 3D map as in other methods [6].
Secondly, we add a spatial context module to the encoder, drawing from projection-based methods like SalsaNeXt [2]. This module comprises small kernel convolutions (with a kernel size of 1 or 3), facilitating the model to propagate global patterns through an increased number of channels.

## 3.2 Auxiliary Task

We propose an auxiliary task of learning point-wise true (de-skewed) height relative to the ground. This task is particularly useful in unstructured environments, where the terrain is not flat and the LiDAR scans are not perfectly aligned with the ground plane. We posit that learning to predict the true heights as an auxiliary task results in the encoder learning features robust to domain-shifts.

Predicting ground-relative height for each point is intuitively a regression task, but to allow for more stable training, we formulate the auxiliary training objective as regression through classification, training the model to predict each point's true height using discrete height bins. This transforms the continuous regression problem into a discrete one, reducing the impact of outliers and noise to improve convergence [12], and allowing for an estimation of confidence in the prediction.

The classification head for the auxiliary decoder therefore estimates the distribution of the true height of each point in the point cloud across the height bins. The loss for the auxiliary task is computed as:

$$ \mathcal{L}_{aux} = \frac{1}{N} \sum_{j=1}^{N} \|\hat{y}_j \cdot h - y_j\|_2^2, \qquad (1) $$

where N is the number of points in the point cloud, $\hat{y}_j \in \mathbb{R}^k$ the prediction from the auxiliary head, $h \in \mathbb{R}^k$ is the vector of centroids of the height bins and $y_j$ the ground truth.

The combined training loss for the model is given by:

$$ \mathcal{L} = \mathcal{L}_{seg} + \lambda \mathcal{L}_{aux}, \qquad (2) $$

where $\mathcal{L}_{seg}$ is the segmentation DICE Loss [13] and $\lambda$ is a hyperparameter that controls the weight of the auxiliary task in the overall training loss.

# 4 Experiments

## 4.1 Datasets

We test our methods on the standard outdoor 3D navigation dataset, Rellis-3D [14], which consists of LiDAR scans from an Ouster OS164 LiDAR sensor mounted on a Clearpath Warthog UGV.

We study the performance of our method by comparing its performance on the target sequences when trained on a source sequences from Rellis-3D, which differ in terms of terrain as well as weather and environmental conditions under which the data was collected. Specifically, we train on sequences with sunny conditions on the trail with bushes and sparse trees and test adaptability to sequences with cloudy conditions on a pasture and forest trail.
<table><caption>Table 1: Comparison on F1 Score with SoTA methods. Source and target refer to corresponding sequences from Rellis-3D [14]. Our model performs best on the source and shows the smallest drop in performance from source to target.</caption><thead><tr><th>Method</th><th>Source</th><th>Target</th><th>Drop</th><th>Params</th></tr></thead><tbody><tr><td>PointNet++ [4]</td><td>0.59</td><td>0.44</td><td>25.42%</td><td><strong>0.9M</strong></td></tr><tr><td>SqueezeSegV3 [3]</td><td>0.71</td><td>0.51</td><td>28.16%</td><td>9.2M</td></tr><tr><td>SalsaNeXt [2]</td><td>0.73</td><td>0.62</td><td>15.06%</td><td>6.7M</td></tr><tr><td>TE-Next [7]</td><td>0.82</td><td>0.66</td><td>19.51%</td><td>5.4M</td></tr><tr><td><strong>Ours</strong></td><td><strong>0.84</strong></td><td><strong>0.72</strong></td><td><strong>14.28%</strong></td><td>5.4M</td></tr></tbody></table>

<table><caption>Table 2: Ablation study showing the percentage improvement in F1 score with the successive addition of each component in our pipeline. Each modification significantly enhances the model's performance.</caption><thead><tr><th>Method</th><th>Improvement</th></tr></thead><tbody><tr><td>Baseline</td><td>-</td></tr><tr><td>+ Augmentations</td><td>13.4%</td></tr><tr><td>+ Spatial Context</td><td>36.5%</td></tr><tr><td>+ Feature Duplication</td><td>59.6%</td></tr><tr><td>+ Auxiliary Task</td><td>67.3%</td></tr></tbody></table>

## 4.2 Implementation details

In our experiments, we evaluate two encoder-decoder backbones to test our proposed modifications. Both backbones are based on Minkowski U-Nets [5]. The first backbone is a simple U-Net architecture inspired by ForestTrav [6], while the second incorporates residual blocks, following the design of TE-Next [7]. Hyperparameter values were tuned manually and fixed for all experiments.

**Labelling for the Auxiliary Task.** We label the true height of each point in the point cloud by estimating the ground plane in each point cloud using the RANSAC algorithm and computing the distance of all points from this estimated plane. We then discretize the height values into *k* bins, each of width 20 cm.

**Augmentations.** We use a combination of augmentations to improve the generalizability of the model. These include small random rotations and jitter. We also add Gaussian noise to the point cloud to promote sensor invariance.

## 4.3 Results

A comparison of our method with current state-of-the-art (SoTA) methods is presented in Tab. 1. Our method outperforms SoTA methods when trained on the same source-target pairings and augmentations in terms of F1 score for both within-domain and out-of-domain segmentation. Source and target domains are identified from the Rellis-3D dataset [14] based on differing terrain and ambient conditions. Our method shows the smallest drop in performance from source to target, indicating its robustness to domain shifts.

Further, we investigate the effect of each of our modifications by running ablative tests on the performance of the model after 3 epochs of training in Tab. 2. Each modification shows a considerable effect on improving the performance of the model.

# 5 Conclusion

This paper introduces a novel approach to improving performance and generalizability in LiDAR-based semantic segmentation for autonomous navigation of unmanned
ground vehicles (UGVs) in unstructured environments. By introducing an auxiliary task of learning point-wise true (de-skewed) height, the model is encouraged to learn features that enhance its robustness to variations in terrain and environmental conditions. Experimental evaluations on the Rellis-3D dataset demonstrate that our method consistently outperforms existing state-of-the-art approaches in terms of segmentation accuracy and domain adaptation. A potential limitation of the method is that the auxiliary task requires labelling through ground plane estimation and is therefore not un-supervised. Future work shall include testing generalization across different datasets collected with different sensors and in more varied environments.

## Acknowledgement

This research was funded in whole or in part by the French National Research Agency (ANR) under the "ANR-23-MOXE-0003" project.

## References

1. X. Xu, L. Kong, H. Shuai, and Q. Liu, "Frnet: Frustum-range networks for scalable lidar segmentation," 2024.

2. T. Cortinhal, G. Tzelepis, and E. E. Aksoy, "Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds for autonomous driving," 2020.

3. B. Wu, A. Wan, X. Yue, and K. Keutzer, "Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud," 2017.

4. C. R. Qi, L. Yi, H. Su, and L. J. Guibas, "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space," 2017.

5. C. Choy, J. Gwak, and S. Savarese, "4d spatio-temporal convnets: Minkowski convolutional neural networks," 2019.

6. F. Ruetz, N. Lawrance, E. Hernandez, P. Borges, and T. Peynot, "Foresttrav: 3d lidar-only forest traversability estimation for autonomous ground vehicles," *IEEE Access*, 2024.

7. A. Santo, J. J. Cabrera, D. Valiente, C. Viegas, and A. Gil, "Te-next: A lidar-based 3d sparse convolutional network for traversability estimation," 2024.

8. C. Saltori, A. Ošep, E. Ricci, and L. Leal-Taixé, "Walking your lidog: A journey through multiple domains for lidar semantic segmentation," 2023.

9. B. Michele, A. Boulch, G. Puy, T. H. Vu, R. Marlet, and N. Courty, "Saluda: Surface-based automotive lidar unsupervised domain adaptation," *Proceedings - 2024 International Conference on 3D Vision, 3DV 2024*, pp. 421–431, 2024.

10. J. Kim, J. Woo, J. Kim, and S. Im, "Rethinking lidar domain generalization: Single source as multiple density domains," 2024.

11. H. Zhao, J. Zhang, Z. Chen, S. Zhao, and D. Tao, "Unimix: Towards domain adaptive and generalizable lidar semantic segmentation in adverse weather," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 14781–14791, June 2024.

12. L. Stewart, F. Bach, Q. Berthet, and J.-P. Vert, "Regression as classification: Influence of task formulation on neural network features," 2023.

13. S. Jadon, "A survey of loss functions for semantic segmentation," in *2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB)*, p. 1–7, IEEE, Oct. 2020.

14. P. Jiang, P. Osteen, M. Wigness, and S. Saripalli, "Rellis-3d dataset: Data, benchmarks and analysis," 2022.