SEMANTIC 3D RECONSTRUCTION USING
SATELLITE IMAGERY

Clément Dechesne, Grégoire Grzeczkowicz, Sara Akodad, Yawogan Jean

Eudes Gbodjo, Bruno Vallet, Pierre-Marie Brunet

► To cite this version:

Clément Dechesne, Grégoire Grzeczkowicz, Sara Akodad, Yawogan Jean Eudes Gbodjo, Bruno Vallet, et al.. SEMANTIC 3D RECONSTRUCTION USING SATELLITE IMAGERY. Big Data from Space 2023 (BiDS), Sergio Albani; Stefanie Lumnitz; Pierre Soille, Nov 2023, Vienna, Austria. pp.125-128. hal-04403446

HAL Id: hal-04403446

https://hal.science/hal-04403446v1

Submitted on 18 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
SEMANTIC 3D RECONSTRUCTION USING SATELLITE IMAGERY

C. Dechesne¹,*, G. Grzeczkowicz²,³,†, S. Akodad⁴,*, Y. J. E. Gbodjo⁴,*, B. Vallet²,*, P.-M. Brunet⁴,\*

¹Magellium, F-31520 Ramonville-Saint-Agne, France

²Univ Gustave Eiffel, ENSG, IGN, LASTIG, F-94160 Saint-Mandé, France

³Direction générale de l'armement, F-75509 Paris, France

⁴Centre National d'Etudes Spatiales, F-31401 Toulouse, France

ABSTRACT

The advent of high-resolution imaging sensors has given rise to in-
novative methods in 3D modeling, remote sensing, and mapping,
particularly in the 3D mapping of urban areas. This paper intro-
duces two novel methods for urban 3D reconstruction using satellite
data, aiming to generate semantic 3D meshes that provide a more
comprehensive understanding of urban landscapes. The first method
employs an object-based reconstruction pipeline, producing an ob-
ject based representation where each object comes with its type and
3D representation. The second technique constructs a single mesh
where each triangle holds a semantic information. We introduce a
new edge-collapse algorithm that incorporates semantic information
during mesh simplification. These methods extend the traditional fo-
cus on buildings to include various urban elements, providing a more
detailed and comprehensive representation of urban environments.
The research contributes to the ongoing development of 3D model-
ing techniques in urban planning and explores new possibilities for
combining geometric and semantic data in 3D reconstruction.

***Index Terms***— 3D reconstruction, remote sensing, semantic meshes, LOD1, Pleiades-HR images.

1. INTRODUCTION

This study is conducted within the French ai4geo scientific and in-
dustrial R&D program involving academic partners (CNES, IGN,
ONERA) and industrial groups (CS Group, AIRBUS Defense and
Space, CLS, GEOSAT, QUANTCUBE). It aims to automatically
generate semantic and geometric information at a global scale from
high resolution satellite imagery. The 3D pipeline involves multiple
steps, as illustrated in Fig. 1.

Initially, the first 3D reconstruction products are a combination
of Digital Surface Model (DSM) and Digital Terrain Model (DTM).
The open-source stereo pipelines CARS and MicMac allows to pro-
duce the DSM from stereoscopic pairs of Pleiades-HR satellite im-
ages. Bulldozer [5] is then employed to obtain a Digital Terrain
Model (DTM) and a Digital Height Model (DHM) from the DSM.
Both DSM and DTM are generated at the same spatial resolution of
the Pleiades-HR satellite images i.e. 50cm.

Using the DTM, ortho-rectified images are computed, and then
processed to obtain semantic maps using deep learning models.
Next, we explore two different ways of combining geometric and
semantic information to produce semantic 3D models. The first
method focuses on above ground objects, taking advantage of a

vectorization of the semantic map and simplifying each object in-
dividually, while the second method treats the scene as a whole
and produces a unique semantic mesh using a new edge-collapse
algorithm.

<figure><img src="image_2.png" /><figcaption>Fig. 1: 3D pipeline</figcaption></figure>

2. RELATED WORK

3D urban reconstruction has traditionally relied on CityGML Level of Detail (LOD) models derived from LiDAR or airborne photogrammetry [4, 9]. Recently, satellite-derived DSM has become popular due to its wider coverage and lower cost. Approaches used can be divided into top-down, utilizing predefined models [4], and bottom-up methods, building structures by simplifying raw data [2].

A possible bottom-up approach is to transform the DSM into a
mesh and then simplify this mesh so that large 3D primitives appear
naturally. A simple and efficient approach of edge-collapse for mesh
simplification has been proposed by Garland and Heckbert in [3]. It
is based on minimizing quadric error metrics to reduce the complex-
ity of triangular meshes while preserving important visual features.

This paper applies Garland and Heckbert's method to various
semantic objects within the urban scene and introduces a new edge-
collapse method that incorporates semantic information globally.
This approach aims to provide a modeling framework that extends
beyond focusing solely on buildings, capturing a more comprehen-
sive representation of the urban environment.

<footnote>*Thanks to BPI for funding</footnote>

<footnote>†Thanks to Direction générale de l'armement for funding.</footnote>
### 3. AI4GEO 3D PIPELINE

Ai4geo's 3D reconstruction techniques involve multiple steps, starting with segmentation using a U-net architecture [11]. Comprising an EfficientNet encoder [13] and a RefineNet decoder [7], the model is trained on Pleiades images with various ground truths, including OpenStreetMap and ai4geo's hierarchical data.

The 3D reconstruction centers around the CityGML Level of Detail (LOD) concept in 3D city modeling. LOD0, the initial 3D modeling stage transitioning from 2D, creates generalized building footprints using different regularization methods (detailed in section 3.1). The MeshCooker technique (section 3.2) then develops LOD1 shapes from LOD0, using the Digital Surface Model (DSM) and Digital Terrain Model (DTM).

We also produce semantic 3D meshes using two techniques. MeshCooker creates object-based semantic meshes for each class of interest using both DSM and DTM (section 3.2.2). Alternatively, we propose to reconstruct all objects at once using only DSM and produce a unique semantic mesh and visually compare both methods (section 3.3).

#### 3.1. LOD0 reconstruction

Once the raster semantic regions have been obtained, morphological post-processing methods, also known as regularization methods, are applied to enhance the shapes of the regions. These techniques aim to refine the object outlines and ensure they conform to regular shapes.

Various methods can be found in the literature to tackle this problem. In ai4geo, three methods were evaluated: Douglas-Peucker [10], which serves as the baseline approach, as well as state-of-the-art methods: ASIP [6] and GeoDan [8]. For more details, the interested reader is referred to [1].

#### 3.2. Meshcooker

Once the land cover raster map has been simplified and vectorized, the main aim is to consider 3D information in order to get higher value products. We developed a Python module called MeshCooker for this purpose. It produces a LOD1 (i.e. independent building with flat roof) and an object-based semantic mesh. The object-based semantic mesh aims at creating a single 3D closed object (i.e. wertight) for each object of the vectorized land cover map.

This can be done by triangulating the DSM and the DTM of the polygon pixelwise and then connecting each surface to each other on the border of the object. Once the 3D object has been created, it can be simplified, using the quadric edge collapse method [3] in order to reduce the number of triangles while preserving visual quality.

##### 3.2.1. LOD1

The LOD1 can be derived from the LOD0 by adding the 3D information extracted from the DSM and the DTM. The idea is to extrude each building according to its height statistic (commonly median or mean). The easiest part is the construction of building walls, since each building of the LOD0 is composed of polygon(s) represented by a list of ordered 2D points. In order to generate ground and roof surface, we need to triangulate all the points of the all the polygon(s) of the building. This can be easily done using the *triangle* python module.

##### 3.2.2. Object-based semantic mesh

The first step for producing the object-based semantic mesh is to mesh the DSM and the DTM independently. This is done for each object of the vectorized land cover map.

Considering georeferenced DSM and DTM of height *I* and width *J* each, let $A_{dsm}$ be a pixel at row *i* and column *j* with height $z_A$. We note $A_{dsm} = (i, j, z_A)$. For each object, we run through all the pixels of the DSM. If the current pixel *A* belongs to the object we want to mesh, 4 other pixels are considered: $B_{dsm} = (i, j + 1, z_B)$, $C_{dsm} = (i + 1, j + 1, z_C)$, $D_{dsm} = (i + 1, j, z_D)$ and $E_{dsm} = (i + 1, j - 1, z_E)$. There are 5 possible triangles we can mesh: ABC, BCD, ABD, ACD and ADE. If one of the point does not belong to the object, we do not consider the triangles composed of this point (e.g. if *E* does not belong to the object, we do not consider the triangle ADE). If *B*, *C*, and *D* belong to the object, we need to choose if we mesh either ABC and ACD or ABD and BCD. The most relevant choice is to keep the edge with lower slope. Therefore, if $|z_A - z_C| > |z_B - z_D|$, we keep the triangles ABD and BCD. If $|z_A - z_C| \le |z_B - z_D|$, we keep the triangles ACD and ABC.

This same process is applied to the DTM also in order to obtain both DSM and DTM meshes for the object of interest. Then, the two meshes are connected so as to acquire a closed meshed object. This latter can then be simplified using [3].

#### 3.3. Unique Semantic Mesh

With Meshcooker, objects are individualized (as connected components of a semantic map) and then treated separately. Another approach to obtaining a semantized 3D model involves meshing the entire DSM. This can be achieved by following the method described in section 3.2.2. Subsequently, each vertex can be labeled using the corresponding pixel in the semantic map. However, this approach results in an extremely large mesh, with twice as many faces as pixels in the DSM.

To fully leverage the advantages of a mesh, which adapts the face size to the complexity of the represented geometry, mesh simplification is necessary. It is crucial for the simplification process to consider not only the mesh's geometry but also the semantic class of each face, which is not the case of traditional edge collapse algorithms such as [3].

To address this limitation, we exploit the fact that our problem can be handled in 2.5D: our inputs are 2D rasters providing a single semantic label and z-coordinate per point. We propose a novel edge collapse algorithm that incorporates both geometry and semantics.

For a face *f*, we define:

* a geometric cost:

$$g_C(f) = \frac{1}{|P(f)|} \sum_{p \in P(f)} |z_{DSM}(p) - z_f(p)|$$

which corresponds to the average over the set $P(f)$ of pixels in *f* of the difference between the altitudes of the DSM and of the face.

* a semantic cost $e_C$ which reflects the entropy of the semantic class labels for the face pixels:

$$e_C = - \sum_j p_j(f) \times \log(p_j(f)) \quad p_j(f) = \frac{|P_j(f)|}{|P(f)|}$$

where $P_j(f)$ are the pixels of *f* with semantic label *j*.
• A combined cost

$$f_C(f) = \lambda \times g_C(f) + e_C(f)$$

Our proposed method involves evaluating the cost of contracting each edge as the difference between the costs of the faces involved before and after contraction. The edges are contracted greedily, starting with the one that has the lowest cost, until a predefined cost limit is reached. With each contraction, the contraction cost of neighboring edges is recalculated to account for the changes made.

Determining the optimal contraction point on the ridge requires a dichotomous evaluation of multiple positions to be contracted. After three iterations, the point that minimizes the contraction cost is selected.

## 4. RESULTS

To evaluate the LOD0 reconstruction quality, several metrics allow a quantitative comparison between ground truth data and produced building polygons. Intrinsic and extrinsic metrics both include

geometric contour (P-cont and R-cont), surface (FNSurf) and topological (RoverSeg and RunderSeg) polygon characteristics detailed in [12]. As explained in [1], experimental comparison, using both metrics and visual results, permits identifying most relevant method which fits better for each considered class. For instance, as GeoDan gives more regular contours, it is considered for the building regularization while Douglas-Peucker algorithm is retained for other classes. This latter vector map is provided as input to the MeschCooker algorithm.

### 4.1. 3D reconstruction

Fig. 2 shows an example of the results obtained with both object-based mesh and unique semantic mesh pipelines.

An area of 4 km² of the Montpellier city center is processed with Meshcooker. It represents around 1500 buildings footprints and 3000 high vegetation footprints. The LOD1 has 86,000 triangles, it is the most simple representation of buildings. However, it might not be relevant for some specific task such as fine 3D simulation. It took around 10 minutes to generate the LOD1. The object-based seman-

<figure><img src="image_3.png" /><figcaption>Fig. 2: Examples of output in Montpellier (France) in different locations. red: building, green: high vegetation, grey: road, purple: rail</figcaption></figure>
tic mesh for buildings has more than 5 million faces and around 2.5
million faces for the high vegetation mesh. It took around 30 min-
utes to generate each class of the object-based semantic mesh. These
representations have already been simplified. The level of the detail
of the output can be adapted to the application by changing the cost
limit of the edge collapse (a higher cost will decimate more, pro-
ducing a more compact representation with less triangles but more
error).

The unique semantic mesh obtained represents a 0.3 km² scene. It took around 4 hours on a single CPU to simplify the entire mesh from 2.752 million faces to 22,000 faces. Qualitatively, λ = 0.5 gives good results. The simplification process can eliminate some of the DSM’s imperfections, but the quality of the final result is highly dependent on the quality of the initial DSM. For example, the DSM obtained from CARS is of rather poor quality for some narrow streets (2e), and since no heuristics are used during simplification, such as forcing building facades to the DTM, the facades of these streets were almost entirely absent (2h).

## 4.2. Discussion

The two methods presented here merge geometric and semantic in-
formation to build a 3D model. The results obtained depend largely
on the quality of the DSMs, DTMs and semantic maps used.

The object approach takes full advantage of LOD0’s regulariza-
tion brick to obtain fairly simple objects. In addition, it forces fa-
cades to be vertical at building contours and to drop all objects on the
DTM. The result is a model that looks much cleaner, and a method
that is more robust to DSM errors (where only the top of objects is
impacted).

On the other hand, the single semantic mesh obtained provides a
continuous representation of the entire scene. The result is necessar-
ily a 2-manifold, and there can be no intersection between objects.
Nevertheless, it is impossible to volumetrically separate entities with
such a representation. Moreover, the absence of a ground class in
the semantic map means that facades cannot be correctly labeled.
For example, it is not possible to differentiate a vertical floor from a
building facade. What's more, it is impossible to label facades when
two objects of different types (e.g. a building and high vegetation)
are located next to each other.

Alternatively, the unique mesh method takes advantage of the
entire classification to also show a segmentation of the ground, while
the object method focuses only on buildings and high vegetation.
Processing objects such as roads or rails at object level is not straight-
forward. What do you do with a road on a bridge, which by defini-
tion would be well above the DTM (2i)? The single mesh approach
doesn't necessarily solve the problem either, because in this case it
doesn't show the geometry of the underside of the bridge (2l).

5. CONCLUSIONS AND PERSPECTIVES

By utilizing satellite imagery, the process of generating 3D meshes
becomes more accessible and efficient. These meshes serve as a
crucial foundation for various simulation applications, such as ur-
ban planning, disaster management, and virtual reality experiences.
Furthermore, the proposed methods are highly adaptable regarding
considered classes. Therefore, DSM and DTM used as inputs signif-
icantly impacts the accuracy of the produced 3D models. By lever-
aging accurate DSM and DTM data, the generated 3D meshes will
exhibit improved fidelity and enable simulation applications to de-
liver more accurate and reliable results. Moreover, future work may
focus on the creation of more accurate LOD2 models, which are

characterized by their finer representation of building structures. Fi-
nally, the introduction of Pleiades Neo images may hold the potential
to enhance mesh precision and accuracy with its high resolution of
30 cm.

REFERENCES

[1] Sara Akodad, Pierre Lassalle, Bruno Vallet, Etienne Le Bihan, and Pierre-Marie Brunet. Ai4geo: Lod0 generation for 3d building models. In *2023 Joint Urban Remote Sensing Event (JURSE)*, pages 1–4, 2023. doi: 10.1109/JURSE57346.2023.10144155.

[2] Ksenia Bittner, Pablo D'Angelo, Marco Körner, and Peter Reinartz. Dsm-to-lod2: Spaceborne stereo digital surface model refinement. *Remote Sensing*, 10(12), 2018. doi: 10.3390/rs10121926.

[3] Michael Garland and Paul S. Heckbert. Surface simplification using quadric error metrics. In *Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques*, SIGGRAPH '97, page 209–216, USA, 1997. ACM Press/Addison-Wesley Publishing Co. doi: 10.1145/258734.258849.

[4] Shengxi Gui and Rongjun Qin. Automated lod-2 model reconstruction from very-high-resolution satellite-derived digital surface model and orthophoto. *ISPRS Journal of Photogrammetry and Remote Sensing*, 181:1–19, 2021. doi: 10.1016/j.isprsjprs.2021.08.025.

[5] D. Lallement, P. Lassalle, Y. Ott, R. Demortier, and J. Delvit. Bulldozer: An automatic self-driven large scale dtm extraction method from digital surface model. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, XLIII-B2-2022:409–415, 2022. doi: 10.5194/ isprs-archives-XLIII-B2-2022-409-2022.

[6] Muxingzi Li, Florent Lafarge, and Renaud Marlet. Approximating shapes in images with low-complexity polygons. In *2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 8630–8638, June 2020. doi: 10.1109/CVPR42600.2020. 00866.

[7] Guosheng Lin, Anton Milan. Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 5168–5177, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society. doi: 10.1109/CVPR.2017.549.

[8] Chris Lucas and Tom van Tilburg. Building boundary. Geo-Dan, 2019. URL https://github.com/Geodan/building-boundary.

[9] Ravi Peters, Balázs Dukai, Stelios Vitalis, Jordi van Liempt, and Jantien Stoter. Automated 3d reconstruction of lod2 and lod1 models for all 10 million buildings of the netherlands. *Photogrammetric Engineering and Remote Sensing*, 88(3):165–170, 2022. doi: 10.14358/PERS.21-00032R2.

[10] Urs Ramer. An iterative procedure for the polygonal approximation of plane curves. *Computer Graphics and Image Processing*, 1(3):244–256, 1972. doi: 10.1016/S0146-664X(72)80017-0.

[11] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In *Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015*, volume 9351, pages 234–241, Cham, 2015. Springer International Publishing. doi: 10.1007/978-3-319-24574-4_28.

[12] Beril Sirmacek, Hannes Taubenbock, Peter Reinartz, and Manfred Ehlers. Performance evaluation for 3-d city model generation of six different dsms from air- and spaceborne sensors. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 5(1): 59–70, 2012. doi: 10.1109/JSTARS.2011.2178399.

[13] Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In *Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning*, volume 97 of *Proceedings of Machine Learning Research*, pages 6105–6114. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/tan19a.html.