<figure><img src="image_1.png" /></figure>

HAL
open science

# Learning to translate land-cover maps

Luc Baudoux

► To cite this version:

Luc Baudoux. Learning to translate land-cover maps: Several multi-dimensional context-wise solutions. Signal and Image Processing. Université Gustave Eiffel, 2021. English. NNT: tel-03977658

HAL Id: tel-03977658

https://hal.science/tel-03977658

Submitted on 7 Feb 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

Université
Gustave Eiffel

# Learning to translate land-cover maps:
several multi-dimensional context-wise solutions

## Thèse de doctorat de l'Université Gustave Eiffel

École doctorale n° 532, Mathématiques, Science, et Technologie de l'Information et de la
Communication (MSTIC)

Spécialité de doctorat : Signal, Image, et Automatique

Unité de recherche : Laboratoire des Sciences et Technologies de l'Information Geographique (LASTIG),
IGN.

Thèse présentée et soutenue à l'Université Gustave Eiffel,
le 10/12/2021, par :

## LUC BAUDOUX

### Composition du Jury

<table><tr><td><strong>Begüm Demir</strong><br>Professeur, Université de technologie de Berlin, Allemagne</td><td>Présidente du jury</td></tr><tr><td><strong>Yuliya Tarabalka</strong><br>Chargée de recherche, LuxCarta, France</td><td>Examinatrice</td></tr><tr><td><strong>Alexis (Lex) Comber</strong><br>Professeur, Université de Leeds, Royaume-Uni</td><td>Rapporteur</td></tr><tr><td><strong>Germain Forestier</strong><br>Professeur, Université de Haute-Alsace, France</td><td>Rapporteur</td></tr></table>

### Encadrement de la thèse

<table><tr><td><strong>Clément MALLET</strong><br>Docteur, Université Gustave Eiffel/IGN/ENSG, France</td><td>Directeur de thèse</td></tr><tr><td><strong>Jordi INGLADA</strong><br>Docteur, CNES/CNRS/IRD/INRAe/UPS, France</td><td>Co-Directeur de thèse</td></tr></table>
"The best translations are always the ones in the language the author can't read." - Jorge Amado
Acknowledgments

The great Jacques Rouxel once said, "It is better to mobilize your intelligence on bullshit than to mobilize your bullshit on intelligent things". Although I have often found myself doing the opposite over the past three years, I have always been able to count on the wise comments and advice of my two PhD advisors to get me back on the right track. I, therefore, address my heartfelt gratitude to Clément Mallet and Jordi Inglada for their unwavering ability to mobilize their intelligence on my bullshit.

I am happy to have been able to share these three years with all of my young engineering and researcher colleagues in the lab: Charly, without whom my incessant tea breaks would have seemed very lonely, but also Anatol, Azelle, Damien, Emile, Ekaterina, Florent, Gregoire, Helen, Medhi, Melvin, Romain, Solen, Teng, Yanis and Yizi with whom I had the chance to share beautiful moments of daily life. Although I cannot name them all because there are too many of them, I would also like to thank all of the "old" fellow researchers, especially Loic, with whom I was lucky enough to share an office until COVID and these long months of remote-working and Laurent for being such a good loser in our competition for whoever arrives the earliest at work.

Of course, I would like to thank IGN, CESBIO and ANR for making this thesis possible. Nonetheless, I am also thankful to the ENSG and its teaching team for allowing me to experiment with my teaching ability on friendly, motivated and, above all, unable to escape students.

Finally, the greatest Professor of all time, Albus Dumbledore, once said, "Happiness can be found, even in the darkest of times, if one only remembers to turn on the light". Thanks to my family and friends for having been this light and, in particular, to Serge, who was my beacon in the darkest hours of this thesis.
Résumé

La description de la couverture biophysique des surfaces terrestres, appelée occupation du sol, est d'une importance capitale dans de nombreux domaines, allant de l'urbanisme aux études climatiques en passant par la sécurité alimentaire. Historiquement produites à la main, les cartes d'occupation du sol ont profité de l'essor de l'imagerie satellitaire et des méthodes avancées de vision par ordinateur pour gagner en précision et en fréquence de mise à jour. Elles souffrent toutefois de deux inconvénients limitant leur utilisation. D'une part, la résolution spatiale des cartes produites est fixe. Or une carte d'une résolution de 10 mètres ne conviendra pas à l'analyse de phénomènes à grande échelle, ni à l'étude d'objets de moins de 10 mètres. D'autre part, la nomenclature de la carte est choisie pour répondre à un besoin spécifique qui ne correspond pas nécessairement aux besoins d'un autre utilisateur. Ainsi, une carte peut regrouper sous le terme "bâti" un ensemble d'éléments tels que des "routes" et des "habitations", qui dans d'autres nomenclatures seront classés séparément.

Les approches actuelles de traduction de nomenclatures sont principalement fondées sur des méthodes de traduction sémantique (LCCS...) appliquées au niveau de la nomenclature en comparant les définitions de classes (la classe "blé" sera traduite en "herbacée"). Ce faisant, elles négligent le fait que deux objets de la même classe peuvent être traduits différemment en fonction, par exemple, de leur contexte spatial ou de leur évolution temporelle. En outre, la traduction de la résolution spatiale est généralement traitée distinctement de la traduction de nomenclature alors que ces deux notions sont intimement liées (un arbre seul ne peut pas être traduit en "forêt").

Cette thèse aborde ce problème en proposant des méthodes de traduction contextuelle augmentant les possibilités de réutilisation et de génération de nouvelles occupations des sols. Dans un premier temps, nous proposons différentes stratégies, principalement fondées sur des réseaux de neurones à convolution apprenant à traduire une carte source en une carte cible en fonction du contexte. Nous montrons l'importance cruciale du contexte spatial et géographique (une forêt en montagne est probablement constituée de conifères) sur de multiples exemples de traductions. Dans un deuxième temps, partant du constat que les modèles de traduction multi-langues donnent de meilleurs résultats que ceux entraînés à traduire d'une seule langue source vers une seule langue cible, nous proposons un modèle de traduction multi-cartes permettant d'obtenir plusieurs nomenclatures cibles à partir d'une carte source. Nous montrons que ce modèle permet d'obtenir des résultats plus robustes que les modèles entraînés sur une seule traduction, en particulier sur des cartes avec peu d'échantillons d'entraînement. Troisièmement, nous expérimentons différentes configurations de fusion multimodale fusionnant des images satellites (optiques et radar) et des données d'élévation du terrain avec des cartes d'occupation du sol. Enfin, nous définissons la notion et proposons une méthode pour construire un espace de représentation
sémantique commun à toutes les occupations du sol. Nous ne représentons alors plus la traduction comme le passage d'un espace de représentation discret à n classes (une nomenclature) vers un autre espace, mais comme un simple changement d'interprétation d'un espace de représentation sémantique continu commun à toutes les nomenclatures. Nous proposons une première application de la notion d'espace de représentation sémantique à la traduction, en nous concentrant sur la traduction de cartes sources non vues pendant l'entraînement du modèle de traduction. Les codes et jeux de données (France entière, six cartes d'occupation du sol, images satellite, vérité terrain) produits au cours de cette thèse sont rendus accessibles pour la reproductibilité et des comparaisons futures.
Abstract

The description of the bio-physical coverage of the Earth's surface, termed land-cover, is of utmost importance in recent decades in many areas, ranging from urban planning to climate studies and food security. Historically manually produced, land-cover maps now take advantage of the recent boom of satellite imagery and computer vision techniques to gain more accuracy and higher update frequency. However, they still suffer from two disadvantages limiting their use. On the one hand, the land cover map spatial resolution is fixed, while a map at 10-meter spatial resolution will not be suitable for analysing large-scale phenomena, nor for monitoring objects less than 10 meters. On the other hand, the map nomenclature is chosen to meet a specific need which does not necessarily suit another user's needs. For instance, a nomenclature may group under the term "built-up areas" a set of elements such as "roads" and "dwellings", which other nomenclatures may classify separately. Current approaches target to adapt these nomenclatures and spatial resolutions. They are mainly based on pure semantic translation methods (LCCS...) applied at the nomenclature level by comparing class definitions. In doing so, they neglect that two objects of the same class can be translated differently depending, for instance, on their spatial context or temporal evolution. This thesis addresses this interleaved problem by proposing context-wise translation methods to increase re-use possibilities and new land-cover map generation. First, we propose different strategies, mainly based on convolution neural networks, learning to translate a source map into a target map context-wisely. In particular, we show the crucial importance of taking into account spatial and geographical contexts (a forest in the mountains is probably occupied by conifers) on multiple translation cases. Secondly, based on the observation that multi-language translation models provide better results than those trained to translate from a single source language to a single target language, we propose a multi-map translation framework allowing us to obtain several target nomenclatures from a unique source map. We show that this model allows for more robust results than models trained on a single translation, especially on maps with limited training samples. Thirdly, we experiment with different multi-modal fusion configurations merging satellite images (optical and radar) and elevation data with land-cover maps. Finally, we define the concept of, and propose a method to build, a semantic representation space common for all land-cover maps, no longer representing the translation as the transformation from a discrete representation space with n classes (a nomenclature) to another but as a simple change of the interpretation of a continuous semantic representation space shared between all nomenclatures. We propose the first application of the concept of common semantic representation space to translation, focusing on the translation of source maps unseen during the translation model training. The codes and datasets (France-wide, six land-cover maps, satellite imagery, and hand-annotated ground truth) produced during this thesis are also accessible for reproducibility and potential comparison purposes.
# Contents

<table><tr><td>1</td><td>Introduction</td><td>1</td></tr><tr><td>1.1</td><td>Key concepts in Land-cover mapping</td><td>3</td></tr><tr><td>1.1.1</td><td>A brief history</td><td>3</td></tr><tr><td>1.1.2</td><td>Nomenclature</td><td>4</td></tr><tr><td>1.1.3</td><td>Resolution</td><td>6</td></tr><tr><td>1.2</td><td>Problem statement</td><td>7</td></tr><tr><td>1.2.1</td><td>Nomenclature translation issues</td><td>9</td></tr><tr><td>1.2.2</td><td>Spatial resolution translation issues</td><td>12</td></tr><tr><td>1.2.3</td><td>Towards continuous mapping</td><td>14</td></tr><tr><td>1.2.4</td><td>Challenges</td><td>15</td></tr><tr><td>1.2.5</td><td>Problem statement</td><td>16</td></tr><tr><td>1.3</td><td>Contents</td><td>17</td></tr><tr><td>1.3.1</td><td>State of the art</td><td>17</td></tr><tr><td>1.3.2</td><td>Datasets and Evaluation metrics</td><td>17</td></tr><tr><td>1.3.3</td><td>Mono Land-cover map translation</td><td>18</td></tr><tr><td>1.3.4</td><td>Multi-land-cover, Multi-modal translation</td><td>19</td></tr><tr><td>1.3.5</td><td>Building a continuous semantic Land-cover map</td><td>19</td></tr><tr><td>1.3.6</td><td>Publications</td><td>20</td></tr></table>

<table><tr><td>2</td><td>Literature review</td><td>21</td></tr><tr><td>2.1</td><td>Nomenclature Translation</td><td>21</td></tr><tr><td>2.1.1</td><td>Defining the relations between classes</td><td>22</td></tr><tr><td>2.1.2</td><td>Translation based on the semantic relation(s) between classes</td><td>28</td></tr><tr><td>2.2</td><td>Spatial resolution translation</td><td>30</td></tr><tr><td>2.2.1</td><td>Down-resolving land-cover maps</td><td>31</td></tr><tr><td>2.2.2</td><td>Up-scaling: land-cover super resolution</td><td>33</td></tr><tr><td>2.3</td><td>Translation using the four levels of context</td><td>35</td></tr><tr><td>2.3.1</td><td>Spatial context: analysing the spatial relations between objects</td><td>35</td></tr><tr><td>2.3.2</td><td>Geographic context: Tackling specific regional land-cover patterns</td><td>39</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>2.3.3 Temporal context</td>
      <td>41</td>
    </tr>
    <tr>
      <td>2.3.4 Cartographic context: learning with noisy data</td>
      <td>43</td>
    </tr>
    <tr>
      <td>2.4 Robust translation</td>
      <td>44</td>
    </tr>
    <tr>
      <td>2.5 Towards learning a semantically consistent representation of land-cover maps</td>
      <td>47</td>
    </tr>
    <tr>
      <td>2.6 Conclusion</td>
      <td>48</td>
    </tr>
    <tr>
      <td><b>3 Datasets</b></td>
      <td><b>49</b></td>
    </tr>
    <tr>
      <td>3.1 Dataset creation protocol</td>
      <td>49</td>
    </tr>
    <tr>
      <td>3.2 Study area and used land-cover maps</td>
      <td>52</td>
    </tr>
    <tr>
      <td>3.2.1 Study area</td>
      <td>52</td>
    </tr>
    <tr>
      <td>3.2.2 Selection of a restricted number of land-cover maps</td>
      <td>54</td>
    </tr>
    <tr>
      <td>3.2.3 Presentation of the input land-cover maps</td>
      <td>55</td>
    </tr>
    <tr>
      <td>3.3 MLULC challenging characteristics</td>
      <td>58</td>
    </tr>
    <tr>
      <td>3.3.1 Nomenclature translation issues</td>
      <td>58</td>
    </tr>
    <tr>
      <td>3.3.2 Resolution translation issues</td>
      <td>60</td>
    </tr>
    <tr>
      <td>3.3.3 Errors</td>
      <td>61</td>
    </tr>
    <tr>
      <td>3.3.4 Spatial generalization</td>
      <td>62</td>
    </tr>
    <tr>
      <td>3.3.5 Conclusion</td>
      <td>62</td>
    </tr>
    <tr>
      <td>3.4 Ancillary data for enhancing the translation process</td>
      <td>63</td>
    </tr>
    <tr>
      <td>3.4.1 Optical imagery: Sentinel-2</td>
      <td>64</td>
    </tr>
    <tr>
      <td>3.4.2 Radar image: Sentinel-1</td>
      <td>65</td>
    </tr>
    <tr>
      <td>3.4.3 DEM: Alos-World3D</td>
      <td>67</td>
    </tr>
    <tr>
      <td>3.5 Ground truth and quality measurement</td>
      <td>67</td>
    </tr>
    <tr>
      <td>3.5.1 Ground-truth datasets</td>
      <td>67</td>
    </tr>
    <tr>
      <td>3.5.2 Quantitative indices</td>
      <td>70</td>
    </tr>
    <tr>
      <td><b>4 Mono Land-cover map translation</b></td>
      <td><b>72</b></td>
    </tr>
    <tr>
      <td>4.1 Map translation without contextual information</td>
      <td>73</td>
    </tr>
    <tr>
      <td>4.1.1 Motivation</td>
      <td>73</td>
    </tr>
    <tr>
      <td>4.1.2 Baselines implementation</td>
      <td>75</td>
    </tr>
    <tr>
      <td>4.1.3 Results</td>
      <td>79</td>
    </tr>
    <tr>
      <td>4.2 Spatial Context</td>
      <td>83</td>
    </tr>
    <tr>
      <td>4.2.1 Manually defined shape indicators</td>
      <td>83</td>
    </tr>
  </tbody>
</table>
<table><tr><td>4.2.2</td><td>Machine Learnt contextual features</td><td>89</td></tr><tr><td>4.3</td><td>Geographic context</td><td>98</td></tr><tr><td>4.3.1</td><td>Determining a source of geographic context.</td><td>98</td></tr><tr><td>4.3.2</td><td>Incorporating geographical coordinates into a convolutional neural network</td><td>103</td></tr><tr><td>4.4</td><td>Temporal context</td><td>109</td></tr><tr><td>4.4.1</td><td>Multitemporal source translation</td><td>109</td></tr><tr><td>4.4.2</td><td>Temporal gap effects</td><td>113</td></tr><tr><td>4.5</td><td>Cartographic Context</td><td>116</td></tr><tr><td>4.5.1</td><td>Motivation.</td><td>116</td></tr><tr><td>4.5.2</td><td>Idea</td><td>116</td></tr><tr><td>4.5.3</td><td>Experimental Protocol</td><td>117</td></tr><tr><td>4.5.4</td><td>Results.</td><td>121</td></tr><tr><td>4.5.5</td><td>Conclusion.</td><td>122</td></tr><tr><td>4.6</td><td>Conclusion.</td><td>123</td></tr></table>

5 Multi-land-cover, Multi-modal translation 126

<table><tr><td>5.1</td><td>Multi-Land-cover map translation</td><td>126</td></tr><tr><td>5.1.1</td><td>Motivation.</td><td>126</td></tr><tr><td>5.1.2</td><td>Method</td><td>128</td></tr><tr><td>5.1.3</td><td>Results.</td><td>131</td></tr><tr><td>5.1.4</td><td>Conclusion.</td><td>142</td></tr><tr><td>5.2</td><td>Multi-modal land-cover translation</td><td>142</td></tr><tr><td>5.2.1</td><td>Data fusion architecture for translation</td><td>144</td></tr><tr><td>5.2.2</td><td>Remote-sensing data sources</td><td>148</td></tr><tr><td>5.2.3</td><td>Experimental protocol</td><td>149</td></tr><tr><td>5.2.4</td><td>Results.</td><td>150</td></tr><tr><td>5.2.5</td><td>Conclusion.</td><td>153</td></tr><tr><td>5.3</td><td>Conclusion.</td><td>155</td></tr></table>

6 Building a semantically continuous Land-cover representation 157

<table><tr><td>6.1</td><td>Introduction.</td><td>157</td></tr><tr><td>6.2</td><td>What is an ideal semantic representation space ?</td><td>161</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>6.2.1</td>
      <td>Encoding class definitions</td>
      <td>162</td>
    </tr>
    <tr>
      <td>6.2.2</td>
      <td>Computing proximity between classes</td>
      <td>163</td>
    </tr>
    <tr>
      <td>6.2.3</td>
      <td>Constraints specific to the zero-shot use-case</td>
      <td>167</td>
    </tr>
    <tr>
      <td>6.3</td>
      <td>How to build a SRS for Land-cover translation?</td>
      <td>170</td>
    </tr>
    <tr>
      <td>6.3.1</td>
      <td>Using language models to obtain high-dimensional SRS</td>
      <td>171</td>
    </tr>
    <tr>
      <td>6.3.2</td>
      <td>SRS optimisation through dimension reduction.</td>
      <td>179</td>
    </tr>
    <tr>
      <td>6.4</td>
      <td>Applications to Land-cover translation</td>
      <td>184</td>
    </tr>
    <tr>
      <td>6.4.1</td>
      <td>Using SRS for zero-shot source translation</td>
      <td>184</td>
    </tr>
    <tr>
      <td>6.4.2</td>
      <td>Results.</td>
      <td>186</td>
    </tr>
    <tr>
      <td>6.5</td>
      <td>Discussion</td>
      <td>190</td>
    </tr>
    <tr>
      <td>7</td>
      <td>Conclusion</td>
      <td>192</td>
    </tr>
    <tr>
      <td>7.1</td>
      <td>Long story short</td>
      <td>192</td>
    </tr>
    <tr>
      <td>7.2</td>
      <td>Insight on potential improvement for operational use</td>
      <td>194</td>
    </tr>
    <tr>
      <td>A</td>
      <td>Land cover nomenclatures</td>
      <td>230</td>
    </tr>
    <tr>
      <td>B</td>
      <td>Semantic correpondance analysis between maps of the MLULC dataset</td>
      <td>234</td>
    </tr>
    <tr>
      <td>C</td>
      <td>Semantic vs statistic translation</td>
      <td>261</td>
    </tr>
    <tr>
      <td>D</td>
      <td>Class definition</td>
      <td>263</td>
    </tr>
    <tr>
      <td>E</td>
      <td>Confusion matrix of CGLS-LC100, CLC, OSO on our ground truth</td>
      <td>274</td>
    </tr>
    <tr>
      <td>F</td>
      <td>Assessing sampling error uncertainty</td>
      <td>280</td>
    </tr>
    <tr>
      <td>G</td>
      <td>Machine Learning and Deep Learning: short introduction</td>
      <td>282</td>
    </tr>
    <tr>
      <td>G.1</td>
      <td>Machine Learning.</td>
      <td>282</td>
    </tr>
    <tr>
      <td>G.2</td>
      <td>Neural network</td>
      <td>283</td>
    </tr>
    <tr>
      <td>G.3</td>
      <td>Convolution neural networks</td>
      <td>285</td>
    </tr>
    <tr>
      <td>H</td>
      <td>EPI interpretation key</td>
      <td>288</td>
    </tr>
    <tr>
      <td>I</td>
      <td>Map and S1/DEM fusion results</td>
      <td>290</td>
    </tr>
    <tr>
      <td>J</td>
      <td>Viual representation of SRS obtained with BoW and Word2Vec</td>
      <td>291</td>
    </tr>
  </tbody>
</table>
<figure><img src="image_4.png" /></figure>
Introduction

Land-cover (LC) is the "(bio-)physical material over the Earth's surface" [105]. As such, the land-cover is both spatially continuous (can be observed at an infinite number of different scales) and semantically continuous (an infinity of biophysical variables, such as humidity, height, biomass or chemical composition, describes each object). Modelling land-cover information requires using a categorisation method to represent this "infinite information" in a finite set by defining one (or, more rarely, several) scales of observation and a finite number of biophysical categories [321], e.g. forest, water... Land-cover classification aims to obtain this categorisation while preserving geolocation information. The product resulting from this process is called a *land-cover map*. This selection and combination of a finite number of biophysical variables into a finite number of groups, called classes, is referred to as the land-cover map *nomenclature*. Similarly, we speak of spatial resolution to describe the scale of observation adopted.

The knowledge of the elements comprising the soils and the subsoils of the Earth's surface is fundamental in many fields of research [104, 171, 255]. Consequently, over the last 30 years, Land-cover maps have become a mandatory baseline for monitoring the Earth's surface status and dynamics. They are, for example, used in areas as varied as urban planning [188] to better understand the mechanisms of artificialization [356], for the study of the climate since they allows albedo estimations [308] or even food security by estimating the proportion and type of cultivated areas [3]. It is also a large-scale monitoring tool for agricultural and environmental protection policies [64]. For that purpose, despite being a notoriously time-consuming procedure [250], many land-cover map products have been generated, covering the entire Earth's surface multiple times, at several spatial scales, and with various nomenclatures [106]. Therefore, the production of land use maps represents an economic, environmental, and scientific challenge.

Land-cover maps are generally tailored to offer a specific nomenclature and spatial resolution that meet a given user needs [292]. In the following sections, we focus on the challenges associated with the design and use of these maps. In particular, we show the considerable impact the choice of spatial resolution and nomenclature can have on the map's potential applications. From this observation, several approaches, termed land cover translation (or harmonisation), have been proposed during the last 40 years to relax the
constraints related to the choice of a nomenclature and a fixed resolution. The translation
aims to transform a source land-cover map nomenclature and resolution into a target one.

Current approaches, illustrated, for instance, by the well-known LCCS framework [68],
assume that all elements of a given source class have the same possible translation. By
analogy with language translation, we argue that those approaches act precisely like a
*word-by-word* translation, as a given source class is always translated into the same target.
Keeping this analogy, this manuscript focuses on breaking the *word-by-word* translation
paradigm by incorporating context information, *e.g.* a forest in a mountainous area should
not necessarily be translated identically as a forest near the sea. In this perspective,
this manuscript identifies different context elements that can be beneficial for translation
based on local spatial context (*Herbaceous* near *Water* might be *Wetlands*), geographical
location or temporal constraints and propose ways to incorporate them into an operational
translation framework.

Performing a manual analysis of the tremendous number of contextual elements and
determining how to use them to improve translation is a time-consuming procedure not
usable under operational constraints as a new analysis is required for each new source
or target map. Instead, this manuscript explores data-driven strategies in which the
contextual elements and how to use them are directly learned from existing source and
target map samples.

Over the last decade, deep learning has been at the heart of significant advances in various
disciplines, such as computer vision and natural language processing, thanks to its good
results on various data [176] such as images [168] and text [318]. In general terms, deep
learning can be seen as one of the numerous machine learning methods, *i.e.* as a set of
automatic algorithms determining input data characteristics leading to a targeted result.
The main advantage of deep learning methods lies in their ability to self-extract groups of
well-tailored features from the input data to answer the problem [101]. This allows them
to respond effectively to the problem posed without requiring a "manual" definition and
computation of those features. This manuscript leverages this automatic feature extraction
to extract contextual information for translation without explicitly defining the contextual
elements to take into account.

As this thesis is to the best of our knowledge, the first to explores the potential of
modern deep learning methods to perform land-cover map nomenclature and resolution
translation, we first describe in the following sections the precise framing of the land-cover
map translation. In particular, we describe the main characteristics of the land-cover maps
and identify the different contextual information that can be used to achieve accurate
translations. We also formalise the problem of land-cover map translation into three
increasingly difficult problems. Finally, we present an overview of the contributions of this
thesis.
# 1.1 Key concepts in land-cover mapping

## 1.1.1 A brief history

Old maps, such as the 1665 Atlas of Joan Blaeu [25], have always included some land-cover information as geographical markers, *e.g.* after the third forest turn on the right. As current topographical maps, they were designed as an abstract representation of reality in which elements are amplified, simplified or even not represented depending on the intended use of the map. For instance, Joan Blaeu's [25] atlas focuses on the road network and considerably simplifies forest geometry representing only specific ones visible from the road.

It will be necessary to wait for the development of aviation and photography, alleviating the need for the tedious collection of in-situ observations, for land-cover to become a proper cartographic subject resulting in the first real land-cover maps [207], *i.e.* geometrically accurate and spatially continuous land-cover focused cartographic products [192]. Often driven by military needs, those first land-cover focused maps were designed manually by photo interpreters analysing the images collected with a nomenclature focusing on potential obstacles for soldiers [194]. Due to the tedious aspect of this operation, Land-cover maps were mainly produced solely on a small spatial extent and never updated. Note that photo interpretation is still used nowadays for its outstanding quality results, which are difficult to match even with the current state-of-the-art image automatic analysis [296]. It not only provides some of the land-cover maps considered as references [118] but is also used for the constitution of data sets used for the automation of land-cover production and validation [189].

In the 1970's, the rise of computing science and the first satellite images [73, 81] launched the era of automatic remote sensing, *i.e.* the production of these maps by algorithms analysing images. The first algorithms were either based on physical modelling [146], requiring to manually define explicit sets of rules to translate an image pixel into a class ("a pixel with a high blue value is probably water") or on statistical resemblance analysis [35]. In parallel, the field of machine learning, which we can define as algorithm learning to "automatically improve through experience" [217], emerged and progressively replaced explicit sets of physical rules with implicitly learnt ones. The machine learning algorithms used for remote sensing mainly belong to the supervised learning paradigm. They learn the features responsible for class assignment on pre-existing pairs of image/map examples. Once trained, the algorithm can be applied to produce maps on new images. The machine learning community referred to the transformation of the almost continuous image information into a set of discrete classes as classification [166]. Therefore land-cover mapping is often termed land-cover map classification. These automated methods exhibited the advantage of their processing speed. They allowed the analysis of ever-larger geographical areas announcing the arrival of the first global-scale maps [191].

In recent years, the exponential improvement of satellite image resolution fostered the
emergence of a new paradigm replacing the independent per-pixel classification with a
neighbour-aware pixel classification. This new paradigm called semantic segmentation,
combined with the arrival of ever more efficient learning algorithms enabled the emergence
of highly accurate maps with rich nomenclatures, high spatial and temporal resolution [334].
However, these improvements are made at the cost of an ever greater need for training
data, particularly by analysing large time series of multi-spectral images via ever more
resource-intensive algorithms [239]. As a result, producing land-cover maps remains a
tedious process, based on advanced technologies and implying significant investment [203].

Ensuring that each map can be used for the highest possible number of applications
is essential. From this observation rose a new field of study between the '70s and the
beginning of the century: Land-cover map standardisation [119]. Standardisation aims to
codify the nomenclature conception by using a shared vocabulary to facilitate the interoperability between LC. Considerable research was conducted on standardisation resulting
in various solutions such as the famous Land-Cover Classification System (LCCS) [67], its
improvement Land Cover Meta Language adopted as (ISO: 19144-2), and more recently,
the EAGLE project [9]. However, none of those frameworks has been universally adopted
by the remote sensing community for many reasons detailed in [54, 145]. Amongst those
reasons, one of them is that, although indisputably necessary, nomenclature standardisation mainly focuses on methods applicable when creating new maps. They often do not or
only partially adapt to the pre-existing maps and does not address resolution translation.

This thesis focuses on increasing the potential use of land-cover maps by presenting
translation methods adapting the resolution and nomenclature of existing maps. The
following sections introduce the notions of nomenclature and resolution and explain all
the related issues.

1.1.2 Nomenclature

**General definition** A map's nomenclature (often termed classification system/class set/classification/legend) describes the categorisation of the infinite set of existing objects into a finite number of classes. Each class is described by a textual definition, which explains its expected content. Let $U$ be the universal set of all possible objects on the ground surface, and let $n_1, n_2, ..., n_m$ be a nomenclature with $m$ classes. The formulas below summarise some essential properties of the nomenclature.

Equation 1.1, referred as the completeness formula, states that the union of all the classes
inside a nomenclature gives all the possible land-cover map types. Conversely, it ensures
that each existing object on the Earth's surface can be associated with at least one class [79].
In other words, there must be no element that does not fit into the nomenclature. It is
essential to specify that this principle often only applies to objects included in the map's
spatial extent, which makes it challenging to use the nomenclature of a land-cover map
covering a spatial extent A to one covering another spatial extent B. Moreover, some
land-cover maps like Corine land-cover [121] defined their nomenclature some decades
ago and can have difficulty mapping recent land-cover types such as agrivoltaism surfaces mixing solar panels and pastoral activities. This results in potential no-data areas that are not reflected by Equation 1.1.

$$ \bigcup_{i=1}^{m} n_i = U. \qquad (1.1) $$

Equation 1.2, referred to as the unicity formula, states that the intersection of two classes gives the empty set or, in other words, that there should be no overlap between the definitions of the different classes [68]. It guarantees the unambiguity of the attribution of a class or that an object can only belong to a single class. A nomenclature's content stems from a compromise between the user needs and the obtainable accuracy given the algorithms and data used.

$$ n_i \bigcap_{\substack{j=1 \\ i \neq j}}^{m} n_j = \emptyset. \qquad (1.2) $$

Nomenclatures are traditionally organised hierarchically. The first level of the nomenclature has a minimal number of classes that can be divided into other classes when considering further levels (see Figure 2.1).

**Land-cover and Land-use** From the 1920's, we began to build an actual formalisation of the semantic content expected in a land-cover map, among other things, via the work of [266]. Foreseeing the economic potential of this type of mapping, particularly in terms of agricultural taxation, the author proposed to distinguish land-cover (the biophysical categories mentioned above, e.g. "forest") from land use (the anthropic use of land-cover, e.g. "forestry") which he believed to have more potential use cases. This is the first of many works to raise this distinction which agitated the world of cartographers and then the remote sensing community throughout the century [192]. Even though the transition from one to the other may seem trivial at first glance, the complexity stems from the multitude of possible associations between the cover and use [193]. For instance, grass-occupied soil can have various associated uses, like agricultural or recreational use (stadium) or even no use (natural meadow). [80] note that while most maps produced between the early XXth century and the 1970's were land-use oriented, ulterior maps were more land-cover oriented. They link this phenomenon to the evolution of mapping methods and, in particular, the arrival of the first low-cost, large-scale satellite images (Landsat-1) and the rise of computing, making possible their automatic interpretation. Their low spatial resolution compared to aerial images complicated land-use inference, which generally requires an analysis of the textural aspect of the image. Consequently, a progressive transition between land-use and land-cover mapping is observed due to the impossibility of determining the corresponding use.
The beginning of the XXIst century is marked by considerable improvement in satellite imagery resolution (Ikonos 2001, Quickbird, Spot5 2002), making use and cover analysis possible. Although the mixing of land-cover map and land use terminologies is pointed out as weakening the use of one or the other [53], most maps from then on mixed the two notions. Thus, in its current acceptance, the land-cover map nomenclature includes both the cover and use. The choice of the proportion of incorporated land-use class stems mainly from a compromise between preserving a high accuracy and responding to end users' needs.

### 1.1.3 Resolution

In remote sensing, the term resolution can either designate the: (i) temporal resolution, (ii) spectral resolution (iii) radiometric resolution or (iv) spatial resolution [238]. By analogy, we describe below the notion of resolution for a land-cover map.

**Temporal resolution** Traditionally describes the time step between two image acquisitions over a given area. By analogy, temporal resolution describes the time-step between two versions of a given land-cover map over a given area in this land-cover map oriented PhD manuscript.

**Spectral resolution** Traditionally describes the width and number of bands in the electromagnetic spectrum acquired by the imaging sensor. In this land-cover oriented manuscript, an analogy with nomenclature could be made. However, the remainder of the manuscript never refer to spectral resolution to avoid confusion and improper use of this term.

**Radiometric resolution** Sensitivity to slight energy difference in the image. Not applicable to this land-cover oriented manuscript.

**Spatial resolution** Describes the smallest distinguishable object by the sensor. By extension, spatial resolution is also used to qualify the smallest object visible in an image while being potentially different from the sensor resolution. To clarify the presentation, the term resolution always designate the spatial resolution in the remainder of this PhD manuscript unless mentioned explicitly.

**Land-cover map spatial resolution General definition** Cartographic spatial resolution is commonly defined as the smallest element visible [86] on the map. Unlike printed maps, where spatial resolution is assessed through a cartographic scale, e.g.. 1cm on the map represents 1km in the real world, land cover maps are intended to be used in a digital format to preserve precise geometric information. Consequently, Land-cover map spatial resolution is directly computed on the digital data and traditionally expressed in square meters, e.g.. the smallest element visible in the data is 100m² on the ground.
For a map in raster format, it is regularly associated with the spatial extent covered by one of its pixels. This assumption makes sense on maps resulting from automatic remote sensing. The classification is generally performed at a pixel level fostering a spatial resolution identical to the imagery used. Conversely, it is typically false for maps resulting from photo interpretation as they usually process the classification at an object-level, i.e. on a group of pixels, by manually drawing land-cover areas on images. To ensure both the global consistency of the product and reasonable conception time, they represent land-cover only if it covers a sufficient area, that is, an area superior to a fixed threshold. This threshold is referred to as the minimum mapping unit (MMU) and might differ significantly from the pixel size of the rasterised map version provided [267]. MMU might vary depending on the considered class or the spatial context, e.g. it is often different inside urban areas [322]. MMU can also include more constraints than the simple area threshold, such as width threshold on linear structures [164] or threshold on class proportion. Geometric and semantic information is partially lost due to the MMU, as multiple pixels with different classes might be grouped to achieve a sufficient spatial extent. As this thesis processes rasterised version of maps, we clearly distinguish the pixel resolution from the minimum mapping unit. As a way to simplify future discussion, we use the term *object* to describe either a pixel or a group of pixels. For instance, we state that "the translation is conducted at an object level" to designate that we transform the characteristics of each pixel/group of pixels independently, depending on their specific features. In opposition, we state that a "translation is conducted at a map or nomenclature level" when all objects with the same class are translated the same way.

Those spatial resolutions are strongly correlated to the nomenclature as the observation of a given class can only make sense within a specific range of resolution. For instance, the notion of "individual tree" does not make sense at one km² resolution, while "forest" does. In addition, the resolution of a map strongly constrains its use. As the resolution of the map increases, more phenomena will be identifiable, e.g. observing an urban heat island at a 1km² resolution prevents fine-grain analysis. Additionally, spatial resolution tends to be constrained to the user's needs. Indeed, working with highly resolved maps is difficult when studying large-scale phenomena (such as climate-related topics) since it represents a massive volume of data and can behave like noisy data by adding too much information compared to the actual need.

## 1.2 Problem statement

This section introduces the precise framing in which we address land-cover map translation. Specifically, we provide insight into current issues with land-cover map translation methods and show the different ways in which land-cover map translation can be addressed.

We define land-cover map translation as the procedure aiming to transform simultaneously an existing map resolution and nomenclature into a target one. We argue that translation
is of significant interest for many downstream tasks as suggested by the plethoric number
of papers using it for tasks as diverse as land-cover map fusion [285] comparison [204]
or change detection [135, 208]. We identified the main potential applications requiring
translation in the Figure 1.1 described below:

A The translation of an old source map into the same nomenclature and resolution as a more recent one enables studying land-cover **change detection** on matching classes and resolution.

B Translating a recent source map to the same nomenclature and resolution as an old target provides an **update** version of the old map.

C Translating a high quality land-cover map into a target nomenclature and resolution can provide **validation** samples to evaluate the quality of a target map.

D Translation can be used to **harmonise** multiple source data into a single target nomenclature resolution to achieve a downstream task such as land-cover fusion.

E Translation can be used to **complete** a small extent target map by incorporating the results of the translation of a large spatial extent source land-cover map.

F Translation can be used to **simplify the spatial resolution** of a highly resolved source into a coarser resolved one.

G Conversely, translation can be used to **improve the source resolution**.

H Translation can be used to, **modify the nomenclature** of the source, add new classes, merge some classes.

<figure><img src="image_5.png" /><figcaption>Figure 1.1: Main possibilities for land-cover map translation. (A) change detection, (B) updating, (C) validation, (D) harmonization, (E) completion, (F) spatial simplification, (G) spatial improvement, (H) semantic modification. See [106] for more details about the existing land-cover maps.</figcaption></figure>
All those applications are, of course, also realisable using the traditional image-to-land-
cover classification paradigm. However, we argue that the land-cover translation and
the image classification paradigm are widely different and exhibit different strengths
and weaknesses presented in Table 1.1. The main advantage of translation is to avoid
processing vast stacks of temporal image acquisitions, which is both time and money-
consuming, by leveraging the information already summarised in existing maps. Despite
translation being a complex task (as exposed in the following sections), the state-of-the-
art chapter shows that most works focus on the translation applications rather than
the translation framework itself [344] resulting in low-quality translation. For instance,
proposed translation approaches generally address the resolution and the nomenclature
translation as two distinct problems. However, these notions are intimately intertwined, as
pointed out in the previous section. For clarity, we address the main issues of the current
state-of-the-art solutions separately in the two following paragraphs.

<table><caption>Table 1.1: Comparison between the image to land-cover classification paradigm and land-cover map translation.</caption><thead><tr><th colspan="2"></th><th colspan="2">Image Classification</th><th colspan="2">Land-cover translation</th></tr><tr><th colspan="2"></th><th>Pros</th><th>Cons</th><th>Pros</th><th>Cons</th></tr></thead><tbody><tr><td rowspan="4">Input data characteristics</td><td>Nature</td><td>Images = many geometric and semantic information</td><td>Requires huge temporal and multispectral images stacks and potentially multi modal data</td><td>A single source map resumes a temporal stacks of images</td><td>Discrete and compressed information representation = some information are enhanced but some are laking</td></tr><tr><td>Features</td><td>Temporal, radiometric, texture = common features in computer vision</td><td>Spatial and temporal generalization issues = Domain adaption</td><td>Easy rule based predictions</td><td>Unusual features (Semantic, geometric, neighborhood) = lack of prior works</td></tr><tr><td>Availability</td><td>High, many providers, high temporal resolution</td><td></td><td></td><td>Most maps are rarely updated</td></tr><tr><td>Data volume & storage</td><td>Sensor noise = generally light and random</td><td>High = multi modal, multi-temporal</td><td>light = one or few maps</td><td>Heavy systemic noise (source map errors)</td></tr><tr><td rowspan="2">Output characteristics</td><td>Predictable classes</td><td>Potentially all classes defined by a distinctive temporal, radiometric or texture pattern are predictable</td><td>In practice, land-use classes are often badly predicted, most remote sensed map have less than 20 classes</td><td>Potentially, all classes provided</td><td>Widely depends on the source nomenclature and resolution</td></tr><tr><td>Computation time</td><td></td><td>Long</td><td>Short</td><td></td></tr></tbody></table>

1.2.1 Nomenclature translation issues

Starting from the observation that the previously mentioned nomenclature standardisation
methods failed to unite all remote sensing practitioners under a single standardisation tool
and that those methods are often not applicable to already existing maps, many works
have attempted to propose nomenclature translation approaches to transform a source
nomenclature into a target one [119]. In this section, we briefly introduce their main
limitations to explain the necessity of developing new approaches.

Existing translation methods are based on semantic analysis of class definitions and
operate directly at the map nomenclature level: they seek to associate each source class
with its potential corresponding target class, as shown in a simple version in Figure 1.2.
In a nutshell, those methods generally rely either on an expert knowledge-based analysis,
or on distances computed from an ontological representation of the classes, or on a ratio of
shared features. A comprehensive presentation of how those methods define those semantic
relations is given in Section 2.1.1.2. They often determine multiple possible associations
per class, e.g. a *Forest* can either be translated as *Coniferous* or *Broad-leaved*. However,
they do not define object-level contextual conditions in which one translation is more likely than the other such as "Broad-leaved is a better translation for thin forest on a riverside". Thus, the only translation performed is the strongest one semantically, i.e. the closest in an ontology or the one with the highest number of shared features. This single final association is referred to as a *hard-association* by [180]. By analogy with the field of text translation, we consider these methods comparable to word-for-word translation of dictionaries of two languages. All possible word translations are known, but only the closest semantically is used. This causes multiple problems. First, the closest semantically can be ill-defined, e.g. *Forest* should theoretically be equidistant semantically from *Conifer* and *Broad-leaved*. Secondly, the closest semantically is not necessarily the most observed statistically. For instance, a class *Building* is semantically closer to *Dense urban* than *Sparse urban* as the second one might include many features other than buildings, e.g. gardens. However, as the two translations are meaningful and in order to reduce the error rate, *Building* should rather be translated into *Sparse urban* areas if *Sparse urban* are more frequently observed.

<figure><img src="image_6.png" /><figcaption>Figure 1.2: Example of semantic translation between two nomenclatures. Methods to determine the possible associations are presented in Section 2.1.1.2.</figcaption></figure>

The resulting translated maps are of even lower quality as the number of possible associations for each source class is high. In order to overcome this problem, most of the studies using translation drastically reduce the number of classes in the target nomenclature to maintain good translation quality. To give an order of magnitude of the hard association's effect on the translation quality, we refer the reader to [225] results on the comparison between an expert-based hard-association translation output and the target LC, which resulted in a limited 57% agreement in their use-case.

Fusion of the translation of several source land-cover maps (land-cover fusion) is sometimes used to replace the nomenclature level translation by an object level one. For instance, if the first map translates one pixel into *Conifer* or *Broad-leaved* and the second map translates the same pixel into *Conifer* or *Shrub* the resulting translation should be *Conifer*. The several source map translations are merged at the pixel level. Using the same terminology as [180], we refer to this object-level translation as *soft-association* methods. However, this strategy supposes the availability of several sources with comparable resolutions.

To obtain good quality translations, linguists translate a word differently based on its context of use. By analogy, we propose to determine context elements that allow the translation of two objects of the same class differently. To our knowledge, no work has
been carried out on this subject. We propose below an initial analysis of the types of
context that can be used when translating a land use at a pixel level. We distinguish four
main types:

1. **Spatial context:** Local spatial pattern of the object to translate. It can further be decomposed into two aspects: i) the object's inner characteristics (shape, area, ...) and ii) the relations with close-by neighbouring objects (class, shape, ...) on the map. The notion of "closeby neighbouring" can not be defined precisely as it widely depends on the source map resolution and both the considered source and target/nomenclature. We present in Figure 1.3 some examples of a pixel of a given class.

2. **Geographic context:** Depends on the geographical location of the mapped object. In general terms, it accounts for spatial relationships at wide spatial extents. It can either involve i) the spatial context between the object of the source map for far-away structures, such as "an area of water far from the sea is not a saltwater marsh" or ii) spatial correlation with elements not inside the source map: geo-morphological or climatic features such as a tree in a mountainous area having a high probability of being a conifer.

3. **Temporal context:** Some land-cover classes are partly defined by their temporal patterns. For instance, Corine land-cover *Rice fields* include parcels which "As part of regular cultivation cycle, rice fields are occasionally left fallow for 1-3 years.". Incorporating temporal context implies using multiple source maps. In this example, three annual maps are needed to provide the required temporal context, as there may be no evidence of the land-cover class at a given epoch.

4. **Structured label noise:** Land-cover map often exhibits specific error spatial patterns [241] and specific error distributions. Figure 1.4 displays some of those specific patterns, such as isolated erroneous pixels or salt and pepper aspects on a small area. Instead of neglecting errors present in both the source and target map, one could leverage those specific error characteristics to increase translation quality. For instance, if a source pixel is classified *Water* while the target is *Forest*, the erroneous map might be identified based on the spatial patterns and error distribution of source and target maps. The ratio between the different error patterns can significantly change from one map to another due to changes in the conception method and data. For example, photo-interpreted maps do not exhibit isolated or "salt and pepper" pixels errors but are more prone to big area misclassification than automatically generated ones. For coherence with the three other terms we term Structured label noise as **Cartographic context**.

These different contexts have in common that they are challenging to set out in the
form of explicit rules as they vary according to the intended translation and are plethoric.
However, it seems possible to learn these rules implicitly by training an algorithm to
analyse existing maps.

<figure><img src="image_7.png" /><figcaption>Figure 1.3: Example of various spatial contexts for a single class (Pasture) of the OSO map. Spatial context analysis can be used to determine potential land-cover map classes such as wetlands, or land-use classes such as green urban area.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 1.4: Various spatial patterns for errors in land-cover mapping.</figcaption></figure>

1.2.2 Spatial resolution translation issues

The resolution translation can be either perceived as a super-resolution problem if one
wishes to increase the map's resolution (by analogy with traditional image processing
methods) or as a problem of cartographic generalisation if one wishes to decrease the
resolution. We provide an in-depth analysis of those notions in Section 2.2, while this
section only underline the main issues to tackle.
<figure><img src="image_9.png" /><figcaption>Figure 1.5: Example of a two-class land-cover map downsampling of factor 5. The result of the Nearest neighbour, a Majority voting and a hypothetical spatial context-aware resampling method are compared. Non-spatial context-aware methods i) perform poorly on cells with multiple land-cover map by erasing most of the information ii) do not provide potential new land-cover map classes</figcaption></figure>

Historically carried out using a set of manually implemented rules, cartographic generalisation aims mainly at summarising the information in a map for better visualization [278]. It proceeds through different object operators such as collapsing, simplifying, displacement and exaggeration (e.g. simplifying the object shape) [82]. The traditional cartographic generalisation definition differs from the one expected for land-cover map generalisation by its objective: Land-cover map generalisation does not seek good visualisation or representation properties but to preserve, as much as possible, the geographical location and the semantic content of each object. This prevents the use of operators such as displacement or collapsing that do not preserve the geographical location. On the other hand, Land-cover map generalisation merges objects, including those belonging to different classes, even if it means creating new classes describing mixed land cover, e.g. a class for areas where small pastures and crop parcels are observed in a mosaic pattern. Multiple works have focused on this land-cover map generalisation issue using a tedious definition procedure of sets of
rule-based transformations that need to be redefined from scratch for each end-user needs and sometimes involve manual corrections. Therefore, no generic generalisation method can be applied to any land-cover map [143]. Moreover, most of the time, only the resulting generalised product is provided to users, not the generalisation framework. Consequently, most works involving generalisation tackle the resampling using the deletion operator through simple interpolation strategies such as majority voting or nearest neighbour. A comparison of those simple operators and a potential spatial context-aware resampling method is presented in Figure 1.5. The currently used nearest and majority voting method tend to neglect much land-cover information as the resulting map only accounts for one of the two classes initially present in the cells (*Vegetation* or *Water*). Conversely, by leveraging spatial context, one can achieve a resampling that combines the *Vegetation* and *Water* into new classes accounting for all the information initially included in the map.

Despite numerous works on a satellite image super-resolution for land-cover mapping [149], super-resolution is much less studied using maps as input due to two main reasons. First, defining explicitly rules to achieve super-resolution is a complex task which required significant advances in computer vision to train algorithms to perform it implicitly. Moreover, it involves adding more resolved data than the source map. Current most successful methods use convolutional neural networks to extract contextual information from images to learn to obtain high-resolved maps from coarse ones [202].

### 1.2.3 Towards continuous mapping

In [321], the authors advocate for the end of the land-cover classification paradigm. Making the same observation as the one we presented earlier on the land-cover map inherent continuous nature, they propose to replace classification, which inherently discretises the land-cover in a limited set of discrete classes by an object-based approach. In other words, instead of mapping an area with a single descriptor such as "forest", they propose to map it at an object level with a set of features which could, for instance, include information on the tree cover, tree height, tree essence, and eventual grassland presence. This new paradigm for land-cover mapping encountered considerable success during the past two decades, and multiple projects are now conducted to implement this paradigm in real operational cases such as the Spanish SIOSE [28], the EAGLE concept [8] and more recently, the CLC+ project [247]. This new paradigm gives LULC maps significantly richer information but also suffers a usability gap [353]. Moreover, as the manually defined features are often difficult to predict automatically [354] most of the current object-level approaches rely on photo interpretation.

Driven by recent advances in natural language processing on the semantic representation of words, sentences and concepts, a few works have recently chosen to learn to encode each pixel of images to output a per-pixel semantic representation of the object rather than classes [32]. They demonstrate that it enables to perform zero-shot land-cover map classification [246], i.e. to predict other classes than those used for training. We argue
that those learnt semantic features could be perfectly tailored to replace the difficult
to automatically predict manually defined features mentioned above, especially in the
land-cover translation setup. Indeed, unlike image classification, in which only the targeted
classification can be semantically encoded, in the translation case, the semantic encoding
can both be applied to the input (a land-cover map) and the output (another land-cover
map). Theoretically, this could enable to train a model to translate a set of maps between
each other and then apply this model to translate unseen during training source maps
into unseen during training target maps. This is particularly interesting since it implies
that if a model is trained on a sufficient number of translations, it would then be able
to be applied to any new land-cover map without needing a sample of data for training.
The obtained object-level semantic representation of land-cover (in this case, pixel level)
could then exhibits the same properties as those expected for the object-based paradigm
mentioned previously.

1.2.4 Challenges

Current solutions for land-cover map translation rest on manually defined sets of rules to
perform both the nomenclature and the resolution translation. As underlined previously,
we argue that contextual information can help perform significantly better translations.
Since defining manually defined context rules for each kind of possible class is unfeasible,
we propose to learn those rules directly from sets of examples using deep learning methods.

The main drawback of learning translation from data is that it requires the existence of
target samples for the learning procedure. The first challenge is to propose methods with
either high generalisation ability (perform well on vast spatial extent even when trained
on small ones) or that can train to translate into a target nomenclature without needing
explicit target examples.

As land-cover mapping is nowadays carried out on scales ranging from entire countries to
continents and worldwide, the proposed solutions should lean toward identifying numerous
kinds of context and class spatial repartition. In particular, all experiments are conducted
France-wide (550,000km²) to ensure the robustness of the learnt method.

As such, it involves defining ways to include far-off geographical contexts. This task is
challenging as the convolution neural network used in this manuscript can only process
small spatial extents due to memory limitations. For instance, studying a 25×25km area
with a 10-meter resolved map implies using a 2500x2500 pixel image which is significant
for CNN while small for geographical context.

The method should also be robust to errors in both the source and target maps, as most
current land-cover map exhibits 10 to 30% errors. As seen in Figure 1.4, most errors tend
to be systematic, with some objects in particular spatial contexts being more prone to
errors. Thus the method could theoretically learn to reproduce target errors which is not
the desired output. Independent validation data should be provided to evaluate whether
the method replicates target errors.

Since free Earth observation satellite programs such as the Sentinel mission started only recently (the sentinel mission started in 2015), most previous land-cover maps tend to be produced only a single time or with a significant time gap between each map due to high production cost. Translation methods should therefore be able to learn on pairs of source/target maps with long temporal gaps, increasing further the noise problem.

Translation also presents the problem of high-class imbalance that might be challenging for learning methods. According to Corine land-cover, the European reference for land-cover mapping, arable crops (the primary land-cover type over France) is around 16000 times more prevalent than the less common Corine land-cover classes agro-forestry. The rarer those training classes are in the training data, the harder they are to predict. Nonetheless, obtaining good results on rare classes is crucial for many applications; otherwise, they would not have been included in the source or targeted nomenclature.

Lastly, computational efficiency needs to be evaluated as land-cover mapping methods tend to be applied to sizeable spatial extents using large volumes of multi-temporal and multi-modal data. More specifically, translation methods should partly be assessed based on their performance/complexity trade-off against traditional image-based methods.

### 1.2.5 Problem statement

From the analysis of challenges conducted above, we derive the following problem statement:

> Is machine-learnt context-aware land-cover translation viable to produce high-quality land-cover maps usable under operational constraints?

To answer this general question, we decompose the problems into four sub-questions:

1. How to translate from a source to a target using various context information?

2. Should land-cover translation be cast as an inherently multi-task problem in which one performs multiple source translations into multiple target ones? For which benefit?

3. In which conditions on the nomenclature and resolution of the source and target map should additional data, such as images, be used to improve translation? How shall the additional data and maps be merged?

4. Are class textual definitions an informative tool to enable using a model on nomenclature and resolution unseen during its training?

The following section presents in a condensed way the outline of this manuscript to answer these different sub-problems.
1.3 Contents

The following section reviews the content of this PhD manuscript. We assume that the reader is familiar with traditional machine learning algorithms, such as random forest, Principal Component Analysis (PCA) or t-distributed stochastic neighbour embedding (t-SNE) and deep learning. More specifically, we assume that the following concepts are familiar to the reader: supervised learning with a machine learning model, the main kinds of deep neural nets (perceptrons, convolutional neural networks, transformers) and training losses. We refer the reader to books [27, 101, 155] and available online material<fnref n="1" /> if necessary. We provide for readers who just want a short explanation of key principles, a very concise (thus incomplete) presentation of deep-learning focusing on convolution neural network in Appendix G.

1.3.1 State of the art

In Chapter 2, we extensively review work related to translation. To clarify, we review nomenclature and resolution translation methods separately since most research does so even though, as mentioned earlier, those two aspects are highly intertwined. Section 2.1 reviews works on the nomenclature translation problem. We first present attempts to create universal nomenclatures applicable to all maps that are easily modulable to transform into other nomenclature. Then, we present some procedures to define links between two nomenclatures. Third section focuses on performing nomenclature-level or per-object translation once the class relations are characterised. Section 2.2 reviews work on resolution translation. We first present works focusing on down-resolving maps. Conversely, we then present works focusing on increasing map resolutions. Section 2.3 presents works focusing on how to learn to translate the four-levels of context we identified in section 1.2.1, namely : spatial, geographical, temporal, cartographic. Section 2.4 reviews works focusing on improving translation using multi-task and multi-modal data fusion that inspire the design of our methods for translation. Lastly, Section 2.5 reviews work focusing on how to learn to obtain a continuous semantic representation of a land-cover map to alleviate the need for a sample of existing targets based on semantic encoding.

1.3.2 Datasets and Evaluation metrics

In Chapter 3, we present the different datasets used for our experiments. As no works
previously focused on learning the land-cover map translation, we needed to generate our
training datasets. We publicly released those datasets which can hopefully foster more
research on the translation problem or be used more widely for research on land-cover

<footnote n="1">¹http://introtodeeplearning.com/</footnote>
map fusion and multi-modal data fusion.

* OSO-to-CLC dataset was first released with our first paper on a single source to single target land-cover map translation. It holds 20k tiles of source/target pairs, covering all the 550,000km² of France's mainland territory. We included two dates for each land-cover map (2016/2018 for one and 2012/2018 for the other) and a more than 5000 point manually photo-interpreted ground truth focused on target classes. This dataset has been downloaded around 20 times since its publication.

* MLULC (Multi Land use land-cover map) was released later. While our first dataset could only be used to evaluate the translation from one land-cover map to another, MLULC can be used as a benchmark for multi-source to multi-target land-cover map translation, Land-cover map fusion, and image and map fusion. The dataset still covers the whole 550,000 km² of France's mainland territory. However, it includes six land-cover maps, 10 m Sentinel-2 cloud-free mosaic with visible and near-infrared spectral bands, the ALOS-WORLD3D digital elevation model and the corresponding aspect value, and 10 meter Sentinel-1 GRD with dual polarisation. MLULC was already downloaded more than 190 times at the time of writing.

In the last section, we present the metrics we use to assess all results. More specifically, we present two ways of assessing quality: i) comparison to target and ii) comparison to independent ground truth, the first offering estimation off per-class metrics while the second enables estimation of sensitivity to errors of the target data. We also provide formulas and short descriptions of used metrics.

### 1.3.3 Mono land-cover map translation

Chapter 4 explores the potential of machine-learning based context analysis for land-cover map translation. We work on the most straightforward setup: trying to obtain a single target map from a single source one, which we refer to as the mono-translation setup. We focus on pure land-cover map translation without using additional data such as images to identify the potential of context wise translation solutions. Section 4.1 presents a preliminary study comparing the non-contextual nomenclature level translation with simple context-aware techniques. We first present two traditional translation methods that perform nomenclature and resolution translation separately. Then we also introduce some simple statistical baselines based either on probability modelling or random forest. We show that such an approach performs poorly in many translation cases and presents few experiments on local spatial context. In Section 4.2, we introduce one of our key-contribution, the Asymmetrical-Unet (A-Unet) proposed for spatial context-aware translation. The network is designed to have the following characteristics: i) change the input resolution into the output ii) take into account large spatial context patterns iii) preserve spatial generalisation ability iv) ensure that rare classes are preserved. In Section 4.3, we answer the geographical context problem by adding a small geographical coordinate sub-module
to encode geographical knowledge and demonstrate the potential of this method to take
into account far-off spatial contexts. In Section 4.4, we experiment with the temporal
dimension of translation. We first focus on the impact of noise induced by the temporal
gap between the source and target on the learning results by comparing three different
scenarios. Secondly, we evaluate the potential of the temporal context presented earlier by
using multiple dates on the source map. Lastly, in section 4.5, we present a strategy to
mitigate the impact of target errors during the training procedure.

1.3.4 Multi-land-cover, Multi-modal translation

In Chapter 5, we successively broaden the translation problem to i) perform multi-source multi-target translation and ii) incorporate other types of data. Building a successful multi-land-cover map translation framework is crucial as obtaining multiple nomenclatures and resolutions from a single one is a step towards obtaining a continuum of nomenclatures and resolutions. Furthermore, designing ways to add complementary pieces of information such as optical images or DEM is not only interesting for the translation problem. It also offers an insight into the high complementarity of those products when trying to obtain high-quality land-cover maps. This task exhibits high importance as the fusion between images and land-cover map databases have always been a standard procedure in photo interpretation but is poorly addressed by current automatic methods. In section 5.1, we present the multi land-cover translation network MLCT-Net we design to address the multi-translation task. The network learns to project all maps into a shared representation space before translating to ensure high generalisation ability, even for maps with few training data. We demonstrate that learning a single multi-land-cover map translation model outperforms widely training multiple mono-translation models. Section 5.2 investigates methods to incorporate multi-modal data into the MLCT-net network to improve the result. We thoroughly present experiments for various kinds of data (optical, DEM, SAR) and demonstrate the interest in land-cover map translation for improving current land-cover mapping methods.

1.3.5 Building a continuous semantic land-cover map

Chapter 6 focuses on obtaining a continuum of representation nomenclature and resolution for land-cover maps. We propose to represent each land-cover map class in a continuous semantic space based on class definition, e.g. *Corn* is closer to *Soy* than *Water* in the semantic space. As mentioned earlier, no works have explored this subject from a land-cover map translation perspective. Thus, we take deep care in defining the suitable characteristics for such space and exploring some of the possible applications. In section 6.1, we introduce precisely the yet unexplored in the literature notion of continuous semantic space for land-cover translation. In section 6.2, we describe the expected properties of the semantic space and propose related quality evaluation metrics. In section 6.3, we investigate how
to build this semantic representation by comparing a pre-trained language model with
a specially fine-tuned to land-cover translation one. In section 6.4, we propose the first
application of semantic space to land-cover res-sampling and zero-shot translation.

1.3.6 Publications

Significant parts of the work presented in this PhD manuscript were published in interna-
tional journals and conferences during the completion of the doctorate.

International journals

* Baudoux, L.; Inglada, J.; Mallet, C. "Toward a Yearly Country-Scale CORINE land-cover map without Using Images: A Map Translation Approach." Remote Sens. 2021, 13, 1060., https://doi.org/10.3390/rs13061060

* Baudoux, L.; Inglada, J.; Mallet, C. "Multi-nomenclature, multi-resolution joint translation: an application to land-cover mapping", International Journal of Geographical Information Science, 2022, https://doi.org/10.1080/13658816.2022.2120996

International conferences

* L. Baudoux, J. Inglada and C. Mallet, "Contextual land-cover map Translation with Semantic Segmentation," 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, 2021, pp. 2488-2491, https://doi.org/10.1109/IGARSS47720.2021.9553693

* L. Baudoux, J. Inglada and C. Mallet, "Deep-Learning Based Multiple Land-Cover Map Translation," IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium, 2022, pp. 1260-1263, https://doi.org/10.1109/IGARSS46834.2022.9883056

**Code** In a commitment to reproducible research, we provide our main research code
under free software license in the following repositories:

* https://github.com/LBaudoux/Unet_LandCoverTranslator

* https://github.com/LBaudoux/MLULC
# Literature review

This chapter reviews valuable works to understand the remainder of the manuscript. Land-cover map translation being an understudied subject, some of the hereafter mentioned works belong to diverse scientific fields such as cartography, remote sensing, natural language processing, computer vision and machine learning. Since this PhD thesis is most likely to be read by members of the remote sensing field, we assume good knowledge of related topics from the readers. We already provide the readers' computer vision and machine learning materials in Section 1.3. Additional material is provided each time a concept not commonly found in remote sensing is addressed.

## 2.1 Nomenclature Translation

Nomenclature translation methods aim to transform all classes from a given source nomenclature to a target one. Nomenclature translation is as a two-step procedure which: i) define the relations between the classes of the two nomenclatures and ii) Use those relationships to translate each source class. Let $S$ (respectively $T$) be the source (respectively target) nomenclature defining a set of classes as described in Equation 2.1.

$$ S = \{ S_i, | 1 < i < c_S \text{ with } c_S \text{ number of source classes} \} \quad (2.1) $$

$$ T = \{ T_j, | 1 < j < c_T \text{ with } c_T \text{ number of target classes} \} $$

Each class is described as a set holding all the objects (in this case, pixel $p_k$) belonging to that class; for instance, source class 1 and target class 3 are described in Equation 2.2.

$$ S_1 = \{ p_k, | 1 < k < t_S(1) \text{ with } t_S(1) \text{ total number of pixels with source class 1} \} \quad (2.2) $$

$$ T_3 = \{ p_k, | 1 < k < t_T(3) \text{ with } t_T(3) \text{ total number of pixels with target class 3} \} $$

[335] and [145] distinguish four possible types of correspondence between a source class $S_i$ and a target class $T_j$ namely: the perfect match when the two classes describe the same objects (Equation 2.3), the inclusion of $S_i$ in $T_j$ (Equation 2.4), e.g. $S_i$ is "corn field" and $T_j$ is "cropland", and conversely the inclusion of $T_j$ in $S_i$ (Equation 2.5) or no relation at
all (Equation 2.6).

$$S_i = T_j. \quad (2.3) \qquad S_i \cap T_j = S_i. \quad (2.4) \qquad S_i \cap T_j = T_j. \quad (2.5) \qquad S_i \cap T_j = \emptyset. \quad (2.6)$$

From observation of real land-cover map translation scenarios, we derive one more relation, namely: the partial overlap, i.e. Wetland can partially overlap with Water or Grassland, but is not included them (Equation 2.7 where $S_i^c$ denote objects that are not in class $S_i$).

$$S_i \cap T_j \neq \emptyset \quad \text{and} \quad S_i \cap T_j^c \neq \emptyset \quad \text{and} \quad S_i^c \cap T_j \neq \emptyset. \quad (2.7)$$

Translation from a source class $S_i$ is straightforward in the case of Equation 2.3 and Equation 2.4 as all pixels belongings to $S_i$ also belong to $T_j$. The two other cases are more challenging as some elements of $S_i$ belong to $T_j$ while some do not. The following sections first review how to define those relations between classes or, in other words, how to identify the Equation corresponding to each couple of source/target classes. We then review how to translate using those relations once they are defined.

## 2.1.1 Defining the relations between classes

The following sections review works in chronological order to highlight current nomenclature translation trends. The first section reviews nomenclature standardisation attempts starting in the '70s which proposes to define all land-cover maps with a single nomenclature system to avoid the partial overlap problem. The second section introduces works starting in the '90s on nomenclature harmonisation, i.e. which tries to establish the links between two nomenclatures.

### 2.1.1.1 Nomenclature standardisation

Nomenclature standardisation frameworks, also termed classification systems, aim to provide a single unified nomenclature in which all land-cover map can be expressed. Nomenclature standardisation requires providing an accurate definition of each class content through a set of descriptors (classifiers) [144]. For example, a class Forest is not simply described as "multiple high trees gathered on a wide spatial extent" because the highly subjective terms "multiple", "high", "gathered", and "wide" can be interpreted differently depending on readers field of work or the spatial extent they are mapping.

Table 2.1 reviews some of the most famous standardisation approaches to date. This table does not seek exhaustivity due to the high number of either country or thematic-specific standardisation system but tries to highlight the main trends. We propose to distinguish three main archetypes of standardisation approaches :

* The predefined classes, hierarchical classification system (PDCHCS) represented in Table 2.1 by USGS-LULC, CLC, GEOCOVER, NLCD and NLUD-C: This approach
<table><caption>Table 2.1: Some of the main land-cover standardisation approaches are ordered by year of conception. Blue lines denote the most influential ones (those used for multiple projects). Intended spatial extent plays a crucial role in the nomenclature content, ie. AARS (Asia) include a "Tea" class that is not included in the Corine land-cover (Europe). The intended resolution is important to define the "purity" of the classification for example, the first version of NLCD only requires 20% of the vegetation being cropland to be classified as cropland.</caption><thead><tr><th>Name</th><th>Abbreviation</th><th>Your</th><th>Link with<br>other methods</th><th>Intended<br>Spatial extent</th><th>Intended<br>spatial resolution<br>(min area - width)</th><th>Classification type</th><th>Example of class definitions to describe a wheat crop</th></tr></thead><tbody><tr><td>USGS LULC<br>Classification Systems [6]</td><td>USGS-LULC</td><td>1976 -</td><td>USA originally<br>but applied worldwide</td><td>4ha, 16ha-20ha</td><td>hierarchical-<br>level 1: 9 classes<br>-level2: 37 classes</td><td></td><td>"Cropland and Pasture: include cropland harvested, including bush fruits; cultivated summer-fallow and idle cropland; land on which crop failure occurs; cropland in soil-improvement grasses and legumes; cropland used only for pasture in rotation with crops; and pasture on land more or less permanently used for that purpose."</td></tr><tr><td>CORINE land-cover [121]</td><td>CLC</td><td>1990 -</td><td>Europe</td><td>25 ha - 100ha</td><td>hierarchical:<br>level 1: 5 classes<br>level 2: 15 classes<br>level 3: 44 classes</td><td></td><td>"Non-irrigated arable land: Cultivated land parcels under rainfed agricultural use for annually harvested non-permanent crops, normally under a crop rotation system, including fallow lands within such crop rotation. Fields with spoidal sprinkler-irrigation with non-permanent devices to support dominant rainfall cultivation are included."</td></tr><tr><td>EarthSat GeoCover Land<br>Cover Legend [61]</td><td>GEOCOVER</td><td>1990</td><td>USGS-LULC<br>Worldwide</td><td>900m²-30m</td><td>13 classes</td><td></td><td>"Agriculture, General: All non-rice agricultural fields, both with crop or fallow, highly managed pastures and hay lands (but not grasslands commonly referenced as "rangeland"); complex mosaics of natural vegetation and cropland. Some orchards and tree plantations, such as palm or date plantations, may be included in this category."</td></tr><tr><td>National land-cover Data<br>Classification System [125]</td><td>NLCD</td><td>1992</td><td>USGS-LULC<br>USA</td><td>900 m² - 30m</td><td>hierarchical:<br>level 1: 6 classes<br>level 2: 25 classes</td><td></td><td>"Cultivated Crops: areas used for the production of annual crops, such as corn, soybeans, vegetables, tobacco, and cotton, and also perennial woody crops, such as orchards and vineyards. Crop vegetation accounts for greater than 20% of total vegetation. This class also includes all land being actively tillable."</td></tr><tr><td>National land use/cover<br>database of China [362]</td><td>NLUD-C</td><td>1995 -</td><td>China</td><td>900 m² - 30m</td><td>hierarchical:<br>level 1: 6 classes<br>level 2: 25 classes</td><td></td><td>"Dryland: Cropland for cultivation without water supply and irrigation facilities; cropland that has water supply and irrigation facilities and planting dry farming crops; croplandplanting vegetables; fallow land."</td></tr><tr><td>Land-cover Classification System [68]<br>(AFRICOVER [303])</td><td>LCCS</td><td>1996 -</td><td>Worldwide</td><td>Scale independent</td><td>1 to n classes</td><td></td><td>See Figure 2.2</td></tr><tr><td>International Geosphere-Biosphere<br>Programme-Data and Information System [190]</td><td>IGPB-DIS</td><td>1996</td><td>LCCS</td><td>1km²-1km</td><td>17 classes</td><td></td><td>"Croplands: At least 60% of area is cultivated croplands"</td></tr><tr><td>South African Standard land-cover<br>Classification System [303]</td><td>SASLCCS</td><td>1996</td><td>LCCS</td><td>South Africa</td><td>900 m² - 30m</td><td>hierarchical:<br>level 1: 12 classes<br>level 2: 27 classes</td><td>"Temporary crops: Includes areas currently under crop, fallow land, and land being prepared for planting (excluding timber production). Land is harvested at the completion of the growing season and remains idle until replanted. Physical boundaries are broadly defined: may include small interfield cover types and farm infrastructure."</td></tr><tr><td>Land-cover Working Group [297, 327]<br>(of the Asian Association on Remote Sensing)</td><td>AARS</td><td>1997</td><td>IGBP-DIS</td><td>Asia</td><td>1 km² - 1km</td><td>hierarchical:<br>up to 7 levels<br>up to 61 classes</td><td>"wheat: wheat"</td></tr><tr><td>UMd Global Land-cover Classification [111]</td><td>UMd</td><td>1998</td><td>IGBP-DIS</td><td>Worldwide</td><td>900 m² - 30m</td><td>13 classes</td><td>"Croplands: lands with 80% of the landscape covered in crop-producing fields. Note that perennial woody crops will be classified as the appropriate forest or shrubs land-cover type."</td></tr><tr><td>National Forest Inventory Land-cover<br>Classification Scheme [333]</td><td>NFI</td><td>1999 -</td><td>Canada</td><td></td><td>900 m² - 30m</td><td>hierarchical:<br>up to 64 classes</td><td>"Herb - Graminoids: A Herb polygon with graminoids greater than 50% of the herb cover. Where graminoids are defined as herbaceous plants with long, narrow leaves characterised by linear venation; including grasses, sedges, rushes, and other related species."</td></tr><tr><td>North American Land Change Monitoring<br>System Legend [175]</td><td>NALCMS</td><td>2005</td><td>LCCS</td><td>North America</td><td>900m²/4ha - 30m</td><td>19 classes</td><td>"Cropland: Areas dominated by intensively managed crops. These areas typically require human activities for their maintenance. This includes areas used to produce annual crops, perennial grasses for grazing and woody crops such as orchards and vineyards. Crop vegetation accounts for greater than 20% of total vegetation. This class does not represent natural grasslands used for light to moderate grazing."</td></tr><tr><td>Sistema de Información de Ocupación<br>(del Studio en España~[28])<br>EIONET Action Group on<br>monitoring in Europe~[8]</td><td>SIOSE<br>EAGLE</td><td>2005 -<br>2013 -</td><td>Spain<br>Worldwide</td><td></td><td>Object-oriented<br>Object-oriented</td><td>Object based<br>Object-oriented</td><td>See Figure 2.2<br>See Figure 2.2</td></tr></tbody></table>
first proposed by [6] with the USGS-LULC defines an ontological representation of n
classes that can aggregate multiple classes into a single one as shown in Figure 2.1.
This approach was later extended with IGPB-DIS [190] system and is still used in
many currently produced land-cover maps. The main advantage of this strategy is
its simplicity and the possibility to fuse multiple classes into one to reduce thematic
errors.

* The hierarchically organised classifiers classification system (HOCCS) represented in Table 2.1 by LCCS, IGPB-DIS, SASLCCS, AARS, NFI and NALCMS: This approach first proposed by LCCS [68] avoid defining a fixed set of classes. It first consists of a three-level hierarchical dichotomous nomenclature structure comparable to the PDCHCS-based classification system resulting in 8 classes distinguishing vegetated and non-vegetated (level 1), terrestrial and aquatic (level 2), artificial and natural (level 3) land-cover map types. Then instead of keeping this hierarchical structure, LCCS proposes a modular phase which defines a set of hierarchically organised descriptors such as Life Form, Cover, Height, and Macro pattern (examples of descriptors are presented in Figure 2.2). Those descriptors are intended to be grouped to form various sets of mutually exclusive classes. This represents a significant paradigm shift. Instead of focusing on classes name, this approach focuses on their descriptors. The descriptor set has evolved over time and has been adopted as an ISO standard under Land-Cover Meta Language (LLCS 3.0).

* The object-oriented classification system (OOCs) represented in Table 2.1 by EAGLE and SIOSE: This recent approach is illustrated by the EAGLE [8] project. Like LCSS, this approach focuses on defining land-cover map descriptors. However, these descriptors are intended to be applied at an object level. This enables the creation of maps in which each object is described as a potentially unique set of descriptors instead of a predefined class. Some argue that it is a new paradigm that cannot be perceived as a classification system [321] since there is no class anymore.

Standardisation considerably simplifies the translation by ensuring that the source and
target nomenclature are both a subset version of the standardised one. Indeed, in this
configuration, the previously mentioned partial overlap between two classes is rarely
observed. However, from Table 2.1, we observe that none of the three presented paradigms
has been commonly adopted [41]. A detailed review of the potential reasons for the
non-adoptions of a standardised system is presented by one of the LCCS creators in [145].
To summarise their observation, the main difficulty is that depending on the covered
spatial extent and the thematic purpose, the descriptors of a class can vary profoundly. For
instance, a forest definition often includes descriptors on tree height or coverage or even
tree essence that varies profoundly depending on the geographical area and the intended
usage of the map [42]. Even when using one of the PDCHCS or the HOCCS approaches,
two forests can have widely different definitions, which considerably limits the interest in
standardising in the first place. Moreover, some descriptors can be challenging to determine
for given pixels either due to a lack of information in the raw data or the algorithm used to
make the map [178]. Lastly, to the best of our knowledge, the OOCS approach is currently
only used in operations set up in Spain through the SIOSE project [28] (the EAGLE
concept is not applied yet but is expected to be used for the Corine land-cover map product
CLC+ in 2024). This is mainly explained by the fact that mapping descriptors represent
a tremendous amount of work at an object level (many descriptors per pixel instead of a
single class) when photo-interpreted and the difficulty through automatic image analysis.
The most closely related automatic strategy found in literature focuses on unmixing the
content of each pixel into fractions of the target classes [24, 31, 83, 84]. Those approaches
are comparable to OOCS, for which the descriptors are the target class percentage inside
a given object instead of physiologic/biotic descriptors. Using the same terminology as
[75], we refer to those maps expressed at pixel levels with target class percentage as "fuzzy
land-cover map classification". Conversely, the traditional one class per pixel approach is
termed "crisp land-cover map classification".

<figure><img src="image_10.png" /><figcaption>Figure 2.1: A simplified example of hierarchical classification based on the Corine land-cover 3 level nomenclature. Green illustrates the per-level classification of an oak forest.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Figure 2.2: LCCS standardisation approach with the example of the classification of an oak tree forest. The first phase (dichotomous phase) is similar to the hierarchical classification and enables an 8-class nomenclature at its third level. The second phase describes the class with a custom set of descriptors. Oak forest definition could incorporate a height descriptor stating that an oak tree is between 3 and 20 meters tall. The descriptors are hierarchically organised according to their expected mappability (the difficulty of determination).</figcaption></figure>

### 2.1.1.2 Nomenclature harmonisation

The previous section point out that most land-cover fallow different standardisation method making standardisation based translation difficult. This section investigates nomenclature harmonisation methods that enable associations between classes of different nomenclatures even when they do not fallow the same standardisation method. Many methods have been proposed to find the relations between two classes, most relying on a semantic analysis of the class definition [339].

The simplest solution is to assess the possible association through expert knowledge [234]. When challenging to obtain from the source map classes are targeted, remote sensing data are often integrated to bring additional information on the target class characteristics [1]. This strategy has the advantage of simplicity, albeit not allowing a detailed understanding of the quality of the translation. Moreover it is hardly generalisable as two different experts can provide different associations, especially on nomenclatures with a high number of classes [78] (more than 10).

Numerous more sophisticated solutions have been proposed to automatically estimate the similarity between classes based on semantic similarities [55, 147]. Semantic similarity is a family of metrics assessing quantitatively two class closeness based on their definitions. It can be computed in numerous ways. For instance, one can represent the two nomenclatures into a shared hierarchical nomenclature tree like the one presented in Figure 2.1 and then count the number of links that separate the two classes [220]. The main advantage
of this method is that it ensures symmetric properties for the semantic similarity, i.e. $d(S_i, T_j) = d(T_j, S_i)$. It can also be perceived as a distance measurement used to embed classes in a euclidean representation space [93]. However, it assumes that the source and target nomenclature can both be represented into a single unified hierarchical tree with for each class of Level n+1 only a single super-class at Level n. Based on the observation that this assumption is often not met [123], i.e. a *herbaceous* class might be simultaneously in the super-class natural *grassland*, *crops* or *pasture*, other kinds of shared nomenclature representation structures have been proposed such as semi-lattices [159]. The main advantage of a semantic similarity-based approach is that it defines a correspondence value between each source and target classes [65].

Some recent references in semantic similarity computation are based on using the LCCS standardisation framework [305]. All nomenclatures are first expressed into the LCCS framework in a standardisation step which is challenging in multiple cases [54]. The class association is then determined by computing the percentage of shared descriptors between classes using a simple version of the Tversky loss [313] defined in Equation 2.8 where $s_i$ and $t_j$ denote the descriptors of the class $S_i$ and $T_j$ and $|s_i \cap t_j|$ the number of descriptors shared by the two classes.

$$ Similarity = \frac{|s_i \cap t_j|}{|s_i|} \qquad (2.8) $$

Multiple improvements to this formula have been proposed in the literature. For instance, [76] propose to take into account the descriptors of $S_i$ that are not included in $T_j$ (the dissimilarities) since the original equation only focuses on the shared ones (the similarities) while both information is equally important. Furthermore, [252] proposes assigning different weights depending on the descriptors as they do not necessarily all revert the same importance. Lastly, in the case of hierarchically organised nomenclature, [253, 254] assigns different weights to each feature based on the hierarchical level of the source and target class. An important observation is that those attribute-based semantic similarities measurement does not preserve the symmetric property of ontological-based measurement, i.e. $d(S_i, T_j) \neq d(T_j, S_i)$; thus, they do not correspond to the mathematical definition of a distance metric. From a semantic point of view, this is meaningful, especially in hierarchically organised nomenclatures. For instance, *wheat crops* is very close to *cropland*, semantically speaking, as they all belongs to a cropland. Conversely, *cropland* is not so close to *wheat crops* as all are not *wheat crops*. For a complete review of the different semantic similarity measurements, refer to [271].

Another way of defining a relationship between two nomenclatures is data-driven, with a direct comparison of pairs of maps. Statistical links between the nomenclatures are obtained by analysing spatially co-occurring sources and target classes. A first naive approach uses a confusion matrix between the source and target maps and assigns the most likely class in the target nomenclature to each source class. [56] propose to translate a source map into a coarser resolved target map by using the number and class of the source
pixels inside each targeted segment as information to train a discriminant analysis. More
recently, Latent Dirichlet Allocation has been adopted in [180] as an unsupervised way to
merge statistical analysis with semantic distances between classes. One of the method's
main advantages is that there may be a significant difference between the theoretical
semantic content of a class and its actual content, limiting the real meaning of the measures
of semantic similarities. For instance, suppose that the class *crops* of a source map has
an accuracy of 70% (because of confusion between the two classes *natural* and *cultivated
grasslands*). The semantic definition of the class only accounts for 70 % of the actual
content of the class. Determining these associations on the real content of the target
classes through a data-driven rather than definition-driven approach tackles this issue.
However, it exhibits two main limitations: it is not flexible and result in poor quality
similarity metrics if a huge resolution gap exists between the source and target.

An important observation is that most real case nomenclature harmonisation results in
low semantic similarity scores between a source class and its ideal target [65]. For instance,
[78] computed the semantic similarity between two famous land-cover maps (CLC, the
reference for Europe land-cover and NLCD, the referenced for USA). They conclude that
amongst the 44 classes of CLC and the 21 classes of NLCD, only three semantic similarities
are equal to 1. Thus, most classes exhibit the partial overlap mentioned at the beginning
of the section.

2.1.2 Translation based on the semantic relation(s) between classes

Once the relations between all the source and the target classes are obtained, the translation
from the source nomenclature to the target is performed.

This task is straightforward when class *S*ᵢ has a unique association in the target nomenclature (Equations 2.3 and 2.4). Other cases are far more challenging and are poorly addressed by semantic methods [120, 145, 165]. Two different strategies can be adopted: i) The hard association method, which consists in assigning to each source label its most likely correspondence, and ii) The soft-association method, when one tries to act at the object level by assigning a different label to a source label depending on some object characteristics.

The hard association method, by far the most observed in literature [120, 142, 272, 309, 311], translates each source class into the most similar target class. This approach has a significant limitation: when a class has more than one non-zero semantic counterpart, translating it to the semantically closest class *de facto* ignores all other possible associations. By analogy with the natural language processing field, we compare this approach to a "word by word" translation in which one knows all the possible translations for a word but only assigns the most frequently observed one. Those issues are illustrated by [225] who translated GLC2000 [17] into CORINE land-cover map [121] only using a semantic hard association approach. They obtained a low 57% agreement with the observed correspondence between the two maps.
The soft association method tries to translate each source class into several target classes depending on some contextual information. They either rely on: i) local spatial context summarization [56] or ii) on merging multiple independent land-cover map translation contexts [317]. For example, local context summarisation is illustrated by [56], which performs the translation at an object level (the object is a segment based on the target segmentation). Based on multiple source pixels class distributions inside each segment, one can train a model (a discriminant analysis) to predict the target class. With this strategy, two source pixels with the same class in the source nomenclature can be translated into two different target classes, provided those two pixels belong to two distinct segments. However, this method has many limitations. First, the targeted map must have a coarser resolution than the source one. Moreover, the method assumes that the target map segmentation is available as source input, which might not be the case in an operational scenario. Last but not least, semantic distances are evaluated for a pair of source/target classes and in no way for a fusion of multiple source classes into a single target [65]. Thus it requires relying data-driven strategy to proceed to translation.

The other strategy merges several source maps to obtain a single target. A semantic harmonisation method determines the associations between source and target classes for all maps. Then, a vote is cast at the pixel level to determine the target label according to the pixel composition in the source classes. Multiple methods have been proposed for such a decision: sum [157] or weighted sum [55, 317] of the semantic similarities of source maps. Recently, [180] proposed a hybrid approach, combining the semantic similarities with statistical correspondences between source and target classes. They trained a Latent Dirichlet Allocation model to obtain a discriminant embedding of the spatial co-occurrence between the classes of the two source maps for 300×300 pixel tiles. Once this spatial co-occurrence embedding is obtained, they rely on a simple probabilistic model to predict the target. They assume that the spatial co-occurrence of the classes observed in the 300×300 pixel tiles is statistically informative on the per-pixel translation. For instance, a tile where 80% of pixels labelled $S_i^1$ in source map one and $S_k^2$ in source map two corresponds to the target class $T_j$, should translate all pixels of the labelled simultaneously $S_i^1$ and $S_k^2$ in $T_j$. The authors pointed out that this assumption is not always met, especially when there is a resolution gap between the two source maps, making the method usable only with maps with the same resolutions. Moreover, this soft association model requires the availability of multiple source maps. Therefore, [202] replaced the multiple maps approach with satellite images to perform soft association without using several source maps. The method relies on the idea that one can learn to classify an image into a target land-cover map using only the source map and prior knowledge of the spatial co-occurrence between the source and target classes. Each image's expected target class distribution is estimated using the source map and the co-occurrence matrix. Then a network is trained to predict this expected target class distribution using a Kullback-Leibler divergence loss function. This method suffers two principal limitations. First, it only uses the source map to estimate the target class distribution instead of using it jointly with the image, losing all the location information. Moreover, it assumes that feeding the network with
the expected class proportion for a given image result in a consistent classification. In an informal and simplified way, this can be conceptualised by the idea that if one knows that 20% of pixels should be water and that the remaining are trees, if inside the data 20% of the pixels shares similar features, it seems plausible to classify all blue pixels as water. However, we argue that the higher the number of target classes, the less consistent these assumptions get. Consequently, [202] only presented their results for a target map with four classes (Water, forest, field, and impervious surfaces) with a source map with 20 labels.

To summarise, current nomenclature translation techniques are conducted as a two-step procedure. They first define the relations between each source and target class through semantic or statistical analysis. This is almost straightforward when all maps are standardised from the beginning, but it is otherwise more challenging. Then, they perform either a complex association that gives poor quality results or a soft association that either requires downgrading resolution or merging multiple maps.

These methods all disregard the different contexts mentioned in the section 1.2.1 (spatial, temporal, geographic, cartographic). For example, the class *Grass* of GlobeLand30 [43] can be translated into several CORINE land-cover map classes: *Green urban areas*, *Sports and leisure facilities*, *Pastures*, *Natural grassland* or *Sparsely vegetated areas*. In such cases, an analysis of the local neighbourhood is required—this advocates for integrating the spatial context in the translation task. As in natural language processing (NLP), land-cover map translation involves too many possible contextual configurations and cannot be manually defined. In NLP, this issue is tackled using machine learning procedures on text corpora [330]. Surprisingly, no attempt at land-cover map translation using machine learning-based contextual translation frameworks has been proposed in the literature so far.

## 2.2 Spatial resolution translation

This section reviews current work on changing the resolution of land-cover maps. Changing the resolution of a map is a complex problem that can not be tackled using the same technique as those used for images. Changing an image's resolution is tackled using interpolation techniques that compute local neighbourhood interpolation, i.e. weighted means. This is not possible with maps, as the pixel values denote categorical variables. If a forest is labelled 1, water is labelled 2, and a cropland is labelled 3, the mean between a forest pixel and cropland ($\frac{1+3}{2} = 2$) should not be considered as water. To avoid this problem, most of the land-cover focused works used discrete interpolation methods, either nearest neighbour or majority voting, that account only for a part of the information, as illustrated by Figure 1.5 on page 13. A clear distinction must be made between the terms up/down-resolving and up/down-scaling. Up/down-resolving is the operation to either increase (up) or decrease (down) the number of pixels of a given image. On maps,
this consist of obtaining more/less pixels respectively on the same spatial extent, i.e. an
up-resolve map has a finer resolution than the down-resolve one. Conversely, up/down-
scaling is a cartographic-specific term which consists of either increasing (up) or decreasing
the scale of the map. As large-scale values describe coarsely resolved maps, an up-scale
map has a coarser resolution than the down-scale one (opposite of the up-resolution). For
clarity, we only use the terms up-resolve (obtain a fine resolved map) and down-resolve
(obtain a coarse resolved one). This section review works on down and up-resolving maps
separately as the involved methods are significantly different. We put to the reader's
attention that far more literature on the down-resolving of maps can be found than on
up-resolving, which is the exact opposite of what is observed in images. Consequently, the
up-resolving section mostly mention works on images rather than maps that might need
some adaptations to work on categorical data.

2.2.1 Down-resolving land-cover maps

Down-resolving land cover maps consist in resuming multiple *p*<sub>S</sub> source categorical pixels into *p*<sub>T</sub> target pixels (with *p*<sub>T</sub> &lt; *p*<sub>S</sub>). The different methods proposed in the literature can be categorised according to i) how they aggregate *p*<sub>S</sub> pixels into *M* groups and ii) How they resume the information inside the *M* groups.

Two aggregation strategies are commonly found in the literature: the grid one and the
target segment aware. The grid aggregation strategy consists in packing the p_S pixels
according to a regular grid with p_T cells, as illustrated in Figure 1.5. It is the most
commonly adopted strategy as it does not involve any extra knowledge. Conversely, the
target segment aware aggregation strategy assumes one already has access to the M intended
groups. For instance, cadastral parcels can be used to aggregate multiple source pixels into
a single value per parcel to which they belong. In the land-cover translation scenario, we
do not assume that the target segmentation can be used as an input as it would drastically
reduce the use case. A third and less studied strategy, the characteristics-based aggregation,
groups the p_S pixels according to their characteristics, either semantic [293] (put adjacent
objects with the same class into one group) or geometric [329] (put adjacent objects with
the geometric characteristics like shape or orientation into one group). However, those
methods are usually defined for working on vector databases and are thus ill-defined when
working on images as they induce strong spatial deformations. Conversely, we observe
that context-wise semantic segmentation methods are known for their ability to learn to
predict the target segmentation. We argue that in the specific case of learnt land-cover
translation, one could theoretically achieve a third grouping strategy (neither grid nor
target segment aware) that we term context-aware grouping.

Multiple strategies have been proposed to resume the information on land-cover maps. The
most straightforward strategy adopted in most studies dealing with multiple land-cover
maps with various resolutions is to rely on the nearest neighbour interpolation [286] as
this method is simple and does not require additional information. However, only one
source pixel inside each of the *M* groups is responsible for the final classification; most of the source information is lost. The second most adopted strategy is the majority voting rule, which resumes the content of the *M* groups using the most frequently observed class, better reflecting the actual content of each group. An interesting observation is made by [345], which showed that the map obtained from either nearest or majority rule techniques applied to a fine resolved source map and the classification from a coarse image give significantly different results. They underlined that those techniques do not give satisfactory results as they do not realistically resume the information. They link this problem to the fact that, as the resolution gap between the source and target increases, the majority class tends to represent only a tiny fraction of all the pixels inside one group. For instance, if in every group, the majority class is *S*<sub>i</sub> and represents 25% of pixels inside the group, the resulting majority voting resume all the groups to *S*<sub>i</sub> despite *S*<sub>i</sub> being only 25% of the land-cover observed in the source map. [153] proposed an interpolation method that preserves the per-class areas observed in the source map to partially alleviate this problem. However, this interpolation technique presents the reverse problem. For instance, the group with the highest proportion of *S*<sub>i</sub> will be classified *S*<sub>i</sub>, even if it represents only 5% of the group, if there are a hundred target groups and a class *S*<sub>i</sub> representing 1% of the area in the source map. [291] observes that all those methods completely neglect the noise in the land-cover maps, which can reach 50% depending on the considered land cover and class. They propose to correct the majority voting using the confidence in each source pixel classification (they assume the availability of this confidence matrix).

However, those strategies neglect that when resuming information, one might want to prioritise some information by giving it more weight. For instance, classes such as *Closed forest* are often defined with a statement such as "more than 85% of the area should be covered by trees". Consequently, if the group have 51% of pixels labelled *Closed forest*, the resulting label should not be *Closed forest* even if it is the majority class. This idea of down-resolving a map by applying selective rules depending on some intended characteristics of the down-resolved map closely relates to Cartographic generalization [261]. Those rules detail how to combine a set of operations to transform the resolution of the maps [233]. Many attempts at classifying those operators can be found in the literature [39, 82, 211, 347]. They usually distinguish the operators that modify the attributes (the class) of the object, such as a reclassification operator, from those that modify its geometry, such as a displacement operator, which represents the object slightly out of place. The set of operators used for land-cover map generalisation might vary profoundly depending on the objective of the generalization [143]. In particular, attribute operators are usable in the land-cover translation case as we seek to reclassify data. Conversely, most geometric operators are unusable as we need to preserve the correct geo-location of objects. A recent trend in land-cover map generalisation tends to make the generalisation model more and more aware of the spatial context of an object before generalizing [329], through machine learned strategies [301], in order to improve the classification of heterogeneous areas. Current attempts mainly focus on training deep learning models [276], usually in a self-supervised manner using generative adversarial networks due to the lack of a
clear definition of a correct generalization when one wants to achieve a map with high readability [306]. However, in the specific land cover translation case, this consideration is irrelevant as we do not seek the land cover's readability but to represent the information semantically and geometrically accurately. Moreover, those cartographic generalisation methods are for now only applied to spatially discontinuous objects of a single class, such as roads or buildings [77, 275, 343]. In the land-cover translation, we must necessarily perform transformation on spatially continuous space with multiple classes. For instance, if a forest is adjacent to a crop field, augmenting the size of the forest necessarily reduce the crop field and eventually require reclassification.

### 2.2.2 Up-scaling: land-cover super resolution

Increasing the resolution of land-cover maps is a hot topic that involves numerous studies. However, most focus on how to transform coarse images into finely resolved maps [158, 304, 360] rather than on how to transform coarse maps into fine resolved ones. They either rely on: i) the use of multi-temporal acquisition of images and make use of the slight spatial shift occurring between each acquisition to estimate a super-resolved version of the images and then proceed to classification [184] or ii) on a two-step procedure which first estimates the fraction of each class inside each coarse image pixel using a spectral unmixing method and then use this fuzzy classification to determine a high resolved map [183]. We point out that the current state-of-the-art methods in super-resolution of land-cover map using images rely on deep convolutional neural networks to analyse the spatial context of each pixels [149, 182].

Conversely, very few works directly address how to transform coarse maps (instead of an image) into finely resolved maps. The limited work on the subject mainly comes from the previously mentioned two-step procedure of creating a fuzzy classification from a coarse image and then super-resolving it. This second step, i.e. super resolving a coarse fuzzy classification into a highly resolved crisp classification, closely relates to this PhD manuscript objective of super resolving a coarse crisp classification into a higher resolved one. Indeed, a crisp classification can be seen as a particular case of fuzzy classification where each pixel has a 100% fraction of a single class and 0 for all others. [29] assimilate this problem as an inverse problem [23, 295]. They argue that the forward problem, i.e. determining coarse fractions from a fine resolved crisp land-cover map, is straightforward. Conversely, they insist that the inverse problem is inherently under-determined as multiple plausible solutions can be found. They insist on the need to use prior information that resolve the inherent ambiguity by constraining the number of spatial patterns of classes that can occur at that resolution. Literature can be divided either by the prior information or the algorithm used. Prior information relies most of the time on predefined assumptions on the spatial distribution of the labels, such as the supposition that the output high resolved classification should be the one with the higher maximum class auto-correlation with coarse maps [10, 213, 319] or a mix between ensuring a spatial continuity of same
class object, which is a common assumption [268], and preserving class proportions [298].
More rarely, this preliminary information is directly extracted from images [287]. The
algorithm used for performing the super-resolution also varies profoundly from linear
interpolation [319] to Hopfield networks [298], to genetic programming [213].

Nonetheless, the crisp and coarse classification to crisp and highly resolved classification
task is significantly more challenging than using a fuzzy classification as input as it
increases the inherent ambiguity of the problem, i.e. the problem is less constrained
and thus has more solutions. Instead of giving exact information about the expected
proportion of each class for a given coarse pixel location, only one class is provided. To
the best of our knowledge, the only proposed method classifies a high-resolved image into
a finely resolved land cover using a coarse one during training [202]. As explained in
Section 2.1.2, this works assumes that the transition matrix *M* between the coarse and
high resolved land-cover map is known. Each coarse label *S*<sub>*i*</sub> probability of corresponding
to a fine resolved target *T*<sub>*j*</sub> is known. For a given source map, one can estimate *D* the
expected number of pixels labelled *T*<sub>*j*</sub> in the target map while being labelled *S*<sub>*i*</sub> in the
source map by simply multiplying the *n* number of pixels labelled *S*<sub>*i*</sub> by the probability for
a pixel label *S*<sub>*i*</sub> to be *T*<sub>*j*</sub> in the target map *M*(*S*<sub>*i*</sub>, *T*<sub>*j*</sub>) as *D*(*T*<sub>*j*</sub>|*S*<sub>*i*</sub>) = *n* * *M*(*S*<sub>*i*</sub>, *T*<sub>*j*</sub>). A deep
convolutional neural network is trained to predict this expected target label distribution
using a statistic-matching loss function. However, as pointed out previously, this method
performs poorly as the number of target labels increases and uses the source map only as
a guide for training an image classification procedure.

In a slightly different setup where the source and target maps are continuous (vegetation
height), [198] proposed a solution that alleviates the need to assume prior knowledge of
*M* by replacing the distribution-based loss with a per-pixel loss by assuming that if a
coarse pixel has an *X* value then the global average of all super-resolved pixel inside the
coarse one should be close to *X*. In practice, they process each source pixel value with a
Multi-Layer Perceptron (MLP) and compute the sum of the mean absolute error (MAE)
between multiple corresponding targets predicted values and the coarse source pixel. They
argue that this per-pixels loss function results always in the same solution for one specific
coarse source pixel value independently from its spatial context. Thus, they also propose
to add information on the pixel location by processing the *x* and *y* pixel coordinates in a
separate (MLP) and merging it with the pixel representation. However, their methods
appear challenging in the land-cover setup, as the categorical representation of land-cover
data prevents using their per-pixel MAE loss. Furthermore, it still neglects the geometric
information of the source map by processing only the image. Lastly, choosing a per-pixel
analysis with an MLP prevents the analysis of the spatial context, which appears not
optimal for the super-resolution task. Indeed pixel super-resolution in a land-cover map
should be highly dependent on the surrounding of the considered pixel, i.e. a pixel in a
middle of a forest or on one of its edges should not be super-resolved the same way.

To sum up, this state-of-the-art on-resolution translation pledges for the use of spatial con-
text information to obtain high-quality results. Moreover, Convolutional neural networks
(CNN) achieve the current state-of-the-art results both in the super and down resolution cases. However, as current literature only includes a limited number of works on CNN applied to maps, discussion on the potential and limitation of this technique on the data must be conducted.

## 2.3 Translation using the four levels of context

In the introduction, we identified four relevant context information levels that one could use to achieve high-quality translation (spatial, temporal, geographical, and cartographic). This section focuses on considering four different contexts using a data-driven strategy rather than the earlier rule-based approach. Influenced by the state-of-the-art results obtained by convolutional neural networks, we mainly concentrate on strategies adopted in this field.

### 2.3.1 Spatial context: analysing the spatial relations between objects

We assume that the reader already has knowledge of deep learning and convolution neural networks and do not detail the underlying methods in this manuscript. We suppose that the reader is familiar with supervised training, convolutions and their parameters (size, stride, dilation), convolution neural network and continuous optimization with loss functions. Readers unfamiliar with those topics are preferably referred to [27, 101, 155] and available online material<fnref n="1" />. However, for a concise (3 pages) introduction to the main principle of machine learning and deep learning concepts used in this manuscript, refer to Appendix G.

#### 2.3.1.1 Definition

Spatial context has long been recognised as a significant element in computer vision [15] and, more specifically, in the characterisation of land-cover maps [288]. Two levels of spatial context for a given land-cover map object are identified. The inner spatial context, commonly used in the remote-sensing community through the term (GEographic) Object-Based Image Analysis (GEOBIA or OBIA), increases classification performance by describing the shape characteristics of the object to classify [18]. For instance, a linear-shaped element is probably a road or a river but is unlikely an ocean. Those methods, traditionally used to classify images, consist of a two-step procedure [130]. First, the images are segmented in homogeneous areas using a segmentation algorithm, the homogeneousness criterium being user defined. In the land-cover translation scenario, the first step of this procedure is by nature irrelevant as we already have access to a

<footnote n="1">¹http://introtodeeplearning.com/</footnote>
homogeneous segmentation in the form of contiguous pixels with the same class in the source map. However, one must notice that this source segmentation might significantly differ from the target one resulting in geometric inaccuracies. Secondly, the information on each segment characteristic is computed. Those different characteristics can be grouped into three distinct categories [71]: spectral information (e.g. the average pixel value inside one segment, its standard deviation...), textural information (e.g. spatial patterns of the pixels inside the segment), and shape characteristics (e.g. is the segment wide, linear, circular ...). As we assume in the land-cover translation scenario that the segment holds one homogeneous categorical variable (the same class for all pixels in the segment), the notion of spectral or textural information is irrelevant. Conversely, we present in the Table 2.2 a set of common shape indicators computable on the land-cover translation setup, mainly inspired from [151]. We acknowledge that many other indicators have been proposed but argue that, as underlined by [130] most of them are highly-correlated and, therefore, including all of them does not improve classification accuracy.

<table><caption>Table 2.2: Common shape features used for evaluating the shape of an object mainly inspired from [151].</caption><thead><tr><th>Name</th><th>Formula</th><th>Description</th><th>Helps to discriminate</th></tr></thead><tbody><tr><td>Area</td><td>A<sub>p</sub> = <sup>1</sup>&frasl;<sub>2</sub> &sum;<sub>i=1</sub><sup>n</sup> x<sub>i</sub>(y<sub>i+1</sub> - y<sub>i-1</sub>)</td><td>x<sub>i</sub>: x coordinate of the ith point of the polygon.</td><td>all classes: discriminates small objects from larger ones</td></tr><tr><td>Elongation</td><td>EL = A<sub>p</sub>/P<sub>p</sub></td><td>P<sub>p</sub>: Perimeter of the polygon</td><td>elongated objects: roads, rivers, sand coastlines</td></tr><tr><td>Circularity</td><td>CI = <sup>A<sub>p</sub></sup>&frasl;<sub>A<sub>o</sub></sub> = <sup>4&pi;A<sub>p</sub></sup>&frasl;<sub>P<sub>p</sub><sup>2</sup></sub></td><td>A<sub>cp</sub>: Area of the circle with the same perimeter</td><td>compact objects: circular irrigated crops, some planted forest</td></tr><tr><td>NestedPoly</td><td>-</td><td>Count the number of polygons inside the polygon</td><td>Wide extent object in which other object are inserted: urban areas (parks, road, buidings) and croplands ( 60% of France cover)</td></tr><tr><td>Convexity</td><td>CO = <sup>A<sub>p</sub></sup>&frasl;<sub>A<sub>in</sub></sub></td><td>A<sub>ch</sub>: Area of the convex hull of the polygon</td><td>regular/irregular shaped polygons: a rectangle cropland has CO = 1, conversely a star shaped building have a lower CO</td></tr><tr><td>MBRH</td><td>-</td><td>Minimum bounding rectangle height</td><td>Estimate the shape spread: building have usually small spatial spread compared to Forest or croplands</td></tr><tr><td>MBRW</td><td>-</td><td>Minimum bounding rectangle width</td><td></td></tr><tr><td>MBRArea</td><td>MBRHeight &times; MBRWidth</td><td>Minimum bounding rectangle area</td><td></td></tr><tr><td>MBRFlatness</td><td>MBRH / MBRW</td><td>Flatness of the minimum bounding rectangle</td><td>Complementary with elongation: helps to distinguish ellongated from compact objects</td></tr><tr><td>MBRAngle</td><td></td><td>Angle between the width of the MBR and the North</td><td>Spatially oriented objects: vineyards are often South oriented</td></tr></tbody></table>

Conversely, the outer spatial context, often termed as texture when the considered object is a pixel or as fragmentation when the considered object is a segment, describes the surrounding of the object [16], i.e. this green object is surrounded by green, blue ones. This outer spatial context has been widely studied in images by studying spectral variations between objects. However, to the best of our knowledge, no study has been conducted on describing the outer spatial context directly on a land cover map, i.e. a grassland near a river is probably a wetland.

The last ten years have demonstrated the high ability of Convolutional neural networks (CNN) to automatically directly learn custom context indicators (rather than manually defined ones). Therefore it appears attractive to explore those techniques to perform inner and outer spatial context-wise land-cover translation.

### 2.3.1.2 CNN based spatial context analysis

We highlight that as no previous work has been conducted on applying CNN using land-cover maps as input, we refer to literature on traditional images. Therefore some of the
conclusions obtained in the mentioned papers will not apply to land-cover maps. For instance, CNN are known to be far more sensible to texture than shape information on traditional images [96, 117]. However, they can also extract shape information in a low texture environment [12] such as the one met in land-cover translation.

[315] stated that the number of papers dealing with land-cover classification through deep learning has almost doubled yearly since 2015. More importantly, they pointed out that convolutional neural networks performed better than other methods to learn very different spatial contexts, as demonstrated by the various studies on specific land-cover types such as urban land-use [133], wetland [141], forest [110] or agriculture [169]. Most work focuses on learning spatial context and considers all the exploitable spatial contexts from the inner one to the farthest possible outer spatial context, which has been recognised as helpful in many domains [365]. The maximum distance influencing the classification of a pixel by the network is referred to as the receptive field. According to [197], it consists either of techniques that down-resolve the images before convolution [69], techniques that modify the convolution properties such as dilated convolutions [186] or increasing the depth of the network. Since reviewing all the network architectures would be impossible, we only review the two networks our work is built on.

The first one is the famous U-Net architecture [257] which offers a good compromise between having a large receptive field through downs-resolving while preserving high-resolution information through skip connections [341]. This network is commonly used in the remote sensing community for its lightweight aspect, which enables both rapid training and inference and avoids overfitting on small datasets. We present the U-Net architecture in Figure 2.3.

<figure><img src="image_12.png" /><figcaption>Figure 2.3: Original U-Net-architecture from [257]. The image is progressively down-resolved to ensure large receptive field. Conversely, copy is used to preserve edges informations.</figcaption></figure>

The second, called atrous spatial pyramidal pooling, comes from the deeplabV3 architecture [45] and is inspired by [115]. It proposes producing multiple down-resolved versions of the same feature maps using dilated convolutions. The core idea behind the dilated
convolution is illustrated in Figure 2.4. Instead of performing the convolution on the eight direct neighbours of a given central pixel (like the upper row of the Figure 2.4), the convolution filter is split to still take into account eight neighbours but at an increased distance from the centre pixel (as is in the second or third row of the Figure 2.4). All the outputs down-resolved images are then concatenated and given to another convolution layer.

<figure><img src="image_13.png" /><figcaption>Figure 2.4: Illustration of spatial pyramidal pooling architecture [45]. The input feature map is convoluted with various independent dilated kernels and a mean average filter. The down resolution is achieved through a stride parameter on the convolution represented on this illustration. The different feature maps obtained are then concatenated and linearly combined by a 1x1 learnt kernel.</figcaption></figure>

One primary specificity of learning spatial context for land-cover mapping is that a large network receptive field is insufficient. Indeed, there is no guarantee that all the relevant spatial context is present in the image in the first place. This stems from two phenomena. First, most studies on land-cover mapping focus on a specific spatial extent; information close to but outside the spatial extent is ignored. Secondly, neural networks can only process limited-size images due to memory constraints. The remote sensing community usually uses a patch-based approach to decompose the study area into small patches that are classified separately [277]. For instance, most of them process images from a few dozens of pixels [273] up to a thousand pixels [359] width, which for a 10-meter resolution
image correspond to 10 km. Consequently, the spatial context can not include far-range
information, such as whether the pixel is in a mountainous area.

[7] demonstrated that increasing the receptive field of a network is always beneficial but that the increase of accuracy follows a logarithmic curve. They linked this behaviour to [196] works that demonstrate that more weight is always given to the centre of the receptive field. This appears problematic as far-range spatial context is lowly taken into account. Moreover, far-range context is not directly interesting by itself, *i.e.* knowing the class of an object 20km away from the object we want to classify is not really informative. The potential resides in the combination of all the object classes and spatial arrangement from the closest to the farthest ones, *i.e.* the general characteristics of a wide area around the object to classify, such as geomorphological information (the area is far from the sea and near a mountain) or landscapes (the area is fragmented into small parcels). For this specific context information, there is no real reason to give more weight to the centre of the area. The following section introduces methods to incorporate information on wide-scale spatial context, which we term Geographic context.

## 2.3.2 Geographic context: Tackling specific regional land-cover patterns

We define geographic context as the macro-scale spatial patterns of classes. The spatial distribution of land-cover classes is highly correlated to environmental, climatic and anthropogenic factors [171]. For instance, in France, trees in mountainous areas are likely to be conifers, the crops cultivated in the northern and southern part of the country are different (due to temperature gradient), and far from cities areas are more likely to hold shrubs than close one (fallow croplands). Consequently, a land-cover translation should be aware of this geographic spatial context.

We can distinguish two main strategies in the literature. The first one is defining homogeneous from a land cover point of view spatial units. For instance, [129] proposed to decompose the globe into 61 bio-realms representing unique combinations of biome and biogeographical realm [177], for which the land-cover macro-scale spatial patterns are considered homogenous. Those homogeneous areas are either used to train separate models on different zones like in [139] or to fine-tune multiple local models from a global one [150, 174]. This strategy has the main limitation that it is time-consuming as it requires training/fine-tuning multiple models. Moreover, its performance is highly dependent on how the local areas are defined, *i.e.* Eco-climatic areas used in [139] do not give the same information as the grid area used in [174]. Lastly, [129] observes that, in reality, macro-scale spatial patterns of land-cover do not exhibit sharp divisions but progressive ones, which this strategy does not deal with.

The second strategy is to feed the classification model directly with information on geographic localisation [201]. Very few works have been conducted so far and to the best of our knowledge, no attempts other than our work [19] experimented with this approach
for land-cover map mapping. Attempts in literature mainly concentrate on using the geographic coordinates to produce a coordinate embedding that could later be used for any classification task. For instance, [348] propose to build a vocabulary V = t1, t2, ..., tn consisting of n words extracted from image tags collected worldwide on data sources such as Flickr, Twitter, and Foursquare. Then they proposed a spatial grid-based approach for each cell represented by the tags found in the cell. For instance, if the vocabulary includes three words("water", "Eiffel tower", "glaciers"), a grid cell on the Paris area is encoded as (1,1,0) as there is water and Eiffel tower in Paris but no glaciers. To avoid having two close-by cells with very different embedding, they also include a neighbourhood weighting on the embedding. As they argue that building this per grid cell embedding is tedious, they proposed to train an MLP to predict this embedding directly from the grid cell coordinate. This grid-level semantic embedding can provide helpful information when training a machine learning method. [49], which focuses on image classification based on geotags, established an interesting comparison between using geographic coordinates to either: i) establish a whitelist of possible target labels depending on the location; e.g. was a Swann already observed nearby those coordinates if yes then the Swann is a possible prediction otherwise its not ii) train an MLP to predict a per-label correction from the coordinates, to be multiplied with the per-class prediction obtained with a pre-trained CNN (with fixed weights) applied to the image with the corresponding (the same strategy is found in [200]) or iii) learn the image per-class prediction and coordinate per-label correction simultaneously. They demonstrate that training using the image and the coordinates simultaneously works better on small datasets. Conversely, combining a pre-trained image prediction with a learnt coordinate correction is more efficient on bigger ones. The main potential of [49] is that, unlike other methods, it does not process data according to a predefined grid but directly the coordinates. This considers that macro-scale spatial patterns of land-cover exhibit smooth variation instead of the sharp ones involved by predefined grid systems.

Interestingly multiple other research has been conducted to incorporate 1D location
information (different from 2D geographical coordinates), such as in natural language
processing to add information on word location inside sentences or in 3D protein structure
analysis to add information on the sequential order of amino acid [363]. The current
state of the art approach, relies on a strategy called positional encoding [318] which was
developed to encode word position in sentences. The core idea is that a correct location
encoding should i) be unique for each location ii) ensure that distance in the embedding
space between any two locations (e.g. between the 2nd and 4th word) should be consistent
across sentences with different lengths and iii) Should be bounded to generalize to any
sentences length. They proposed to encode the word position using a set of sine and cosine
functions with varying frequencies before adding them to the network. Let d be the total
number of dimension of a word embedding and k be one of those dimension, then position
encoding is given by Equation 2.9.
$$ \vec{p}_x^{(i)} = f(x)^{(i)} := \begin{cases} \sin(\omega_k x) & \text{if } i=2k \\ \cos(\omega_k x) & \text{if } i=2k+1 \end{cases} \quad \text{where} \quad \omega_k = \frac{1}{10000^{2k/d}} \quad (2.9) $$

As the positional encoding provides a tremendous improvement in the natural language processing field, some works have been conducted to adapt it to work on 2D location information that could be used into CNN. We distinguish two kinds of coordinates that can be position encoded: the intra-image coordinates termed pixel coordinate (the pixel is in the i'th row and j'th column) and the geographical coordinates (this pixel is located at X/Y longitude/latitude). Current work focuses solely on pixel coordinates. They aim to remove the convolution translation invariance [187]. All pixels are submitted to the same convolution when using a CNN, which might be unsuitable in some specific cases, i.e. when working with rasterized wind data, a different part of the images is not exposed equally to wind [314]. [235] propose a strategy encoding image pixel coordinates using a simple adaptation of the positional encoding: rows and columns are independently encoded using Equation 2.9 and concatenated.

In the land-cover translation scenario, we aim to use geographical coordinates. Two key differences are observed with pixel ones. First, the translation invariance should be preserved as all the pixels of one patch belong to the same limited spatial extent; thus have the same geographic context. Consequently, each pixel of a patch should be summed with the same geographical context embedding. Secondly, while pixel coordinates are the same in train and test (there is always a first row/column pixel), the geographical coordinates used are different in training and testing. Thus the effectiveness of positional encoding is yet to be shown in this case as it requires generalisation ability.

### 2.3.3 Temporal context

#### 2.3.3.1 Definition

We define the temporal context of a land-cover nomenclature as all the explicit and implicit temporal patterns exhibited by the classes of the nomenclature. Explicit temporal patterns include all information on temporality included in the class definition. For instance, in the *Rice Field* definition presented in Definition 1 we underlined typical explicit temporal patterns. We observe that inside a definition, multiple temporality levels might be included, "periodically" here describes a temporal pattern occurring multiple times across a year. At the same time, "regular cultivation cycle" and "occasionally left fallow for 1-3 years" describe a pattern with a lower temporality (every three years).

"Rice Fields: Cultivated land parcels prepared for rice production, consisting of periodically flooded flat surfaces with irrigation channels. As part of regular cultivation cycle, rice fields are occasionally left fallow for 1-3 years. These parcels are considered to be rice fields, too." - Corine Land-cover
Implicit temporal patterns include all information on temporality that is not directly included in the class definition but characterises the class content evolution across the land-cover reference period. For instance, a Wheat class inside a map covering an annual period describes a temporal pattern with a period of bare soil followed by wheat growth and another bare-soil period.

Those temporal patterns are directly influenced by the reference period covered by a land-cover map. On the same given spatial extent, a map covering a monthly period distinguish *Bare soils* from *Vegetation*, an annual distinguish *Fallow* from *Annual crops* and *Perennial* ones by studying the variation of vegetation across the year, and a map covering several reference year distinguish *Natural areas* from *Cultivated* ones by studying the vegetation variation across multiple years [221]. Translating often involves matching classes with different temporal contexts, e.g. a *Rice crops* observed during a one-year period, one observed once during a three-year period or a flooded area observed during a one-month period.

Land-cover translation temporal context must be clearly distinguished from the operational temporal gap between the source and the target map encountered when training a machine learning model to perform the translation. We consider this temporal gap as noise-inducing and address it in the following subsection.

### 2.3.3.2 Methods

Historically, classification from multiple images with different acquisition dates is mainly proceeded by concatenating all images into a single one and directly processing the raw data [91, 126]. A common strategy to improve this multi-temporal classification was to complement it with additional spectral features computed separately for each date based on spectral indices such as NDVI [103, 323] or EVI [13, 26] and texture indices [88]. SpectroTemporal aware features were later proposed to reflect the temporal evolution of the data better. Standard spectrotemporal features mainly include statistical metrics (average, min/max, standard deviation of per pixel spectral values on the period) [51, 89, 95, 210], phenology metrics derived from statistical metrics (beginning/end, length, amplitude of the growing season) [148], or on reshaped raw data feature space through dimensionality reduction techniques such as PCA [100]. In the land-cover translation scenario, those spectrotemporal features are not applicable as they were designed to work on continuous feature space, i.e. the average value of a pixel for different reference periods has no real meaning on land-cover maps as it involves doing the mean of categorical variables. Therefore, those works should be adapted to be usable in the land-cover maps translation setup.

The obtained features are historically used to train various machine learning algorithm including decision tree [95, 126], random forest [51, 139], support vector machines [37, 100], Gaussian Maximum Likelihood [148, 249], or multi-layer perceptron [13, 103]. More recent works focusing on the integration of large temporal sequences of data into neural
architectures (RNN [137], LSTLM [273], transformer networks [94]) are considered out of
scope of this manuscript as most land-cover maps are only available in very small temporal
sequences. Indeed most are produced only once or on less than five reference periods.

2.3.4 Cartographic context: learning with noisy data

Translation raises questions about the impact of source and target data errors. The traditional rule-based translation approach *de facto* ignores this problem as (i) the target map is not used in a learning procedure; target map errors can not be learnt (ii) the nomenclature translation is based on source class description, which does not take errors into account. Thus errors in the source map are agnostically translated into target classes. Learning to translate from a source to target behave oppositely. First, some errors in the source maps can potentially be compensated by the translation procedure. For instance if all roads in the source maps are misclassified as urban areas, some of them can be correctly classified as roads in the target nomenclature by using this specific erroneous spatial pattern of linear-shaped urban areas to distinguish roads. Secondly the targeted map errors can influence the translation results. For instance, if all roads in the target map are misclassified as urban areas, the network might learn to replicate the same error by transforming all roads of the source nomenclature to urban areas. Fighting against the effect of label noise is mandatory as the method should be designed to be able to translate between source and target with numerous classes, which tend to be have significant error rates [302].

Learning with noisy data is a highly studied subject. The manuscript only focuses on label
noise (in opposition to image noise). Label noise in land-cover mapping can be systemic or
random depending on the noise source [87]. Most of the systemic noise in land-cover maps
is induced by errors of the classifier producing the maps, which tend to repeat the same
error on similar objects. Therefore it is instance dependent rather than class dependent.
For instance, *Croplands* pixels on the field's edge can be classified as a *Forest*, with this
confusion only happening in this specific spatial setup. Random label noise is more often
observed on photo-interpreted maps, mainly due to temporal gaps between the data used
and the target temporality. In the real case scenario, most errors tend to be observed
between classes semantically close for photo-interpreted land-cover maps or on classes
close in the feature space for automatically derived land-covers. Unlike image noise which
is often considered additive to the true information, label noise behaves as correlated to
original information noise.

[284] propose to classify the different deep learning approach to deal with label noise in four categories: i) architecture-based [336] ii) regularization based [280] iii) Loss based [205] iv) sample selection based [283], which can be combined [364]. Loss-based strategies, which aims to correct the loss computation by assuming some knowledge on the label noise appears particularly relevant in the land-cover translation case as unlike most problems a noise transition matrix is most of the time provided (also termed confusion matrix in
remote sensing). For instance, assuming that the noise transition matrix is known, one
can correct the prediction of the network by multiplying it by its error probability [237].

However, current loss based solutions use transition matrix to correct prediction at a
per-pixel level, e.g. if *Forest* is misclassified in *Water* 20% of the time in the target map,
then noise transition aware loss encourages the prediction on a *Forest* pixel to be 80%
*Forest* 20% *Water*. We believe it corresponds to misuse of the noise transition matrix,
which are class level indicators rather than instance level [46], i.e. one object is 100%
*Forest* or *Water*, and 20% of all objects labeled *Forest* should be *Water*. We underline
that the main reason for the current misuse of the noise transition matrix is that they
were mainly developed to address image classification. In this setup, one image has one
class. Thus, in an iterative algorithm such as those used to train deep learning models,
each iteration only processes a small number of objects per-class. This prevents computing
per-class distribution estimates such as the previously mentioned 20% of all objects labeled
*Forest* should be *Water* as there could be only two images labeled *Forest*. We argue that
in the case of land-cover translation, we could alleviate this limitation as we have access
to the vast number of annotations; land-covers are semantic segmentation of images in
which each pixel has one class. We underline that only a very limited number of works
have been conducted to adapt label noise correction methods to semantic segmentation
maps and that none of them was loss based to the best of our knowledge.

## 2.4 Robust translation

This section focuses on obtaining high-level quality translation models, which we mainly
define as: i) robust to change in the class distribution and spatial arrangement to enable
generalisation on spatial extent not covered initially, ii) able to preserve a maximum of
the target class diversity. Three main research fields have been proposed in the traditional
image to land-cover classification.

First, domain adaptation [312] defines methods able to achieve high-quality results when
the training and testing data exhibit statistically different distributions [167]. Radiometric
values in two different eco-climatic areas can be widely different; the results of a model
trained on one ecoclimatic area might not be transferable to another. In the case of
land-cover map translation, spatial domain-adaptation does not deal with radiometric
values but with class distribution and varying spatial patterns, e.g. fragmented-crop
landscapes might be observed in the training data while open-field is observed in the
testing data. The literature mainly focuses on extracting and projecting features into
a representation space shared between the training and the testing data. In this space,
training and testing data are expected to exhibit the same statistical properties without any
observable shift between them. Traditional methods mainly rely on statistical matching
strategies such as multidimensional histogram matching [138], or principal component
analysis (PCA) [228]. More recent works adopted deep neural networks for their high
generalisation ability [226]. Two constraints are found in the literature to enforce the
training and testing to be mapped into a shared space: (i) minimising the distance in
the representation space between two identical elements through a loss term [231], (ii)
adversarial training [342] in which a discriminator enforces source and target observations
to be comparable. In the land-cover translation scenario, loss based strategy aims to ensure
minimal distance in the representation space for comparable target classes independently
from their nomenclature. For instance, *Forest*, *Coniferous*, *Broad-leaved* or *Shrubs* should
all be represented closely in the representation space, to ensure that a model trained to
translate a source map into a target on area where there is no *shrubs* is usable on an area
with *shrubs*. This shared representation space should not only account for classes but also
spatial context, *i.e.* two elongated forest should be represented closer from one another
than a more circular one. Theoretically adversarial training should obtain the same result
but is confronted with the well-known difficulties in optimising adversarial networks and
are not be explored in this manuscript.

A second strategy is to rely on multi-modal data fusion, which focuses on defining methods
to combine heterogeneous sources of information [98]. Fusing independent sources of
information (such as image, text, video, or audio) describing the same object tends to
enforce learning an object representation less sensitive to local variations and imprecision.
Most current research focuses on merging optical and radar data in land-cover mapping
as they bring complementary features [269]. In the land-cover translation setup, image
and map fusion could enable to learn a more robust representation of the data, enabling
the prediction of higher class diversity. Papers resuming the output of the GRSS data
fusion contest (held each year since 2006 [232]) are a rich source of information on the
current state-of-the-art method for performing a fusion of remote sensing data. Until 2018,
proposed methods relied on extracting manually defined features from the various data,
like NDVI for optical data and class proportion for land-cover data [349] and feeding it
to an ensemble classifier (often random forest). [340] pointed out that the 2018 contest
was the first one in which the best results were always observed using deep learning to
extract features from raw data directly. From then on, even though some slight changes
are observed in the deep learning architecture, the core remains the same. Each data
is encoded using a separate network and merged at a level of abstraction, going from
simply concatenating all raw data to just before a final classification layer. Even though
fusing multiple data at the raw-data level should yield the best inference as it ensures
that no information is lost, [170] observe that various complicating factors such as noise
and contradicting data might benefit from fusing the different data at a higher level
of abstraction [160]. Different works have concluded differently on the best location to
merge data. Choosing the right location to perform data fusion remains highly empirical
and depends on the task and the chosen architecture [134, 244]. Experiments are yet
to be conducted for the land-cover translation setup. A recent trend in multi-modal
data fusion, is to incorporate an attention mechanism [21]. First introduced for natural
language processing, it learns to balance each feature's importance better by analysing all
features simultaneously [48, 318]. Figure 2.5 presents one of the multiple adaptations of
the attention mechanism to images proposed by [358]. The main idea is to process a set of feature maps in three independent feature linear combination layers (f,g,h). The attention map is obtained as the scalar product between transposed f and g. It corresponds to a learnt per-pixel weight. Lastly, the scalar product between this attention map and h is computed to obtain the output of the attention mechanism. Following [199] observation, we underline that the attention mechanism can be used to give selectively more weights to some modalities when merging multiple data sources.

<figure><img src="image_14.png" /><figcaption>Figure 2.5: Per-pixel Self-Attention mechanism for CNN proposed by [358]. Figure reproduced from [358].</figcaption></figure>

The last strategy consists in multi-task learning, which aims to improve inference accuracy on several tasks by training simultaneously on all of them [74], based on the assumption that at least a subset of the task is related [361]. In the image to land-cover classification, common multi-task strategies involve either predicting simultaneously land-cover and land-use [22], land-cover and change detection [63] or land-cover and height [38]. We argue that land-cover translation is an ideal setup for multi-task prediction. The translation could benefit from training a single model to transform one source land-cover into multiple targets. For instance, recent advances in natural language processing have shown that learning single multiple translation models achieved comparable results with learning multiple one-to-one cases. However, the former yields more robust results on languages with few samples and has better generalisation abilities [59]. Multi-language training seems to benefit from the obtained multi-language common representation space [245]. Multitask networks are also commonly trained with the dual objective of reconstructing the input data (self-reconstruction) and achieving the desired tasks [346] to enforce mapping to a shared representation while preserving the unique features of each input.
## 2.5 Towards learning a semantically consistent representation of land-cover maps

We mentioned in Section 1.2.3 that the current trend in land-cover map standardisation is to perform an object-based nomenclature, in which each object is assigned a set of descriptors depending on its characteristics [4]. A similar tendency is observed in the field of zero-shot learning. The core idea in zero-shot learning is to train a network to encode a subset of classes into a consistent semantic space. At inference, classes unseen during training are predicted based on auxiliary information on the links between the classes used for training and the unseen ones. [90] takes the example that with the information "a zebra is more-or-less like a horse but with black-and-white stripes", a child can recognise a zebra without seeing one before, provided he was first taught what a horse and a stripe pattern look like.

We believe that this notion of semantic representation at an object level should be a cornerstone of machine-learnt translation for two reasons: operational and information preservation. First of all, from an operational point of view, working on semantic representation on classes could alleviate the need to retrain the method at each new source or the target map, i.e. once the concept of "forest", "trees", "coniferous", "broad-leaved", "mixed forest", "dense vegetation" have been learnt by the model at various scales and on various spatial extent through the use of multiple sources and target maps it should not be necessary to retrain the model if one wants to translate the concept "woodland". Secondly, from an information preservation point of view, the whole idea of applying a machine learning strategy is to extract contextual information on each object of the source land-cover map to perform better translation than a non-context-wise one. However, while precious, this contextual information is lost once used to assign the target class.

[324] reviews current strategies used for zero-shot learning. They first propose to distinguish them according to the nature of the learnt semantic representation: manually engineered semantic spaces or learnt ones. From a land-cover translation point of view, the LCCS framework presented earlier might be seen as manually engineered semantic spaces in which each class is described by a set of descriptors. They also distinguish them by the nature of the method trained to map into the semantic space. Classifier-based approaches focus on how to directly learn a classifier for the unseen classes, while instance-based focuses on obtaining labelled instances belonging to the unseen classes and using them for classifier learning.

Most works for land-cover map focus on the method to train rather than on the nature of the semantic space. For instance, from a land-cover mapping point-of-view, one of the first works was conducted by [107, 246]. They used a pre-trained word2vec [214] model to translate each class name into a feature vector. They then trained a CNN to obtain these feature vectors from a source image, i.e. instead of training to assign to each pixel a class, such as a forest, they assign the corresponding feature vector. During the test phase, they
rely on a simple approach, such as a k-nearest neighbour, to assign unseen classes.

We believe that more importance should be given to the definition of the embedding space. Indeed the traditionally used word2vec representation of classes has many limitations, namely: i) it only takes into account the class name rather than class description, ii) it is not specialised in land-cover; natural grassland and cropland can be very close in the feature space while we traditionally make a distinction between the two in land-cover mapping.

An important observation is that significant works on label encoding have been conducted during the last decades, especially on how to encode label hierarchy [256], class co-occurrence statistic [212], semantic attributes [172] and subsets of those information [279]. Thus it appears interesting to see how to adapt those works to encode semantically and contextually land-cover maps.

## 2.6 Conclusion

In summary, current land-cover translation solutions address the nomenclature and resolution translation separately. Nomenclature translation solutions are mainly based on semantic associations evaluated at the nomenclature level (LCCS, EAGLE ...). Most of the time, all pixels of a given source class are only translated into a single target (the semantically closest one). Resolution translation is, on the contrary, tackled using either simple interpolation techniques such as the nearest neighbour or majority voting that are not able to infer new classes (a patchwork of water and trees is probably a wetland) or not addressed when it comes to land-cover super-resolution (exception made of [202]). We propose to investigate how to improve translation by performing the nomenclature and resolution translation context-wisely jointly. The currently best-performing methods to incorporate spatial, geographical and temporal contexts replaced manually defined feature-based solutions with machine learnt ones, principally convolution neural networks. As those solutions require training data, we present the dataset used for our experiments in the following section.
# Datasets

This chapter introduces the datasets used during this PhD thesis. Learning the translation requires selecting land-cover maps to train the models. As we concentrate on supervised learning strategies, the introduced datasets propose pairs of the corresponding source and target maps. Since no public dataset with multiple land-cover maps is available, the Multiple Land-use Land-cover (MLULC) dataset, including six land-cover maps, is introduced. A smaller version of this dataset, the OSO to CLC dataset, is also provided to enable specific experimentation. The slight variations between the two datasets are explicitly mentioned when appropriate. Section 3.1 introduces the different steps to create a land-cover translation dataset. Criteria influencing the choice of the study area and the choice of the used land-cover maps are emphasised. Section 3.2 introduces the study area and the used land-cover maps main characteristics. Section 3.3 presents the differences between the land-cover maps and underlines the main challenges of the dataset. Section 3.4 introduces additional data (optical images, synthetic aperture radar images and digital elevation models) provided with the dataset that can be used to improve the translation quality. Finally, Section 3.5 presents a manually built ground-truth dataset that enables fair assessment of the quality of the translation by avoiding quality evaluation based on a comparison to a noisy target reference. This last section additionally reviews the metrics used in this PhD manuscript to assess the quality of the translation.

## 3.1 Dataset creation protocol

This section introduces the different operations used for the dataset creation and highlights the elements that influence the choice of the study area and land-cover maps. The dataset creation procedure involves seven steps presented in the following subsections.

**Download** Land-cover maps can be distributed in two different formats: raster or vector. Raster format (i.e. image format) describes the land cover using a grid, in which each cell is represented by a pixel value. This format is usually the one used by land-cover maps stemming from automatic image classification. Conversely, vector format describes the land-cover using a set of spatially contiguous polygons, often obtained by manual
delineation by photo-interpreters. As this manuscript focuses on convolutional neural
network application, raster format is the final delivery format of our dataset. However,
when possible the land-cover are dowloaded in vector format as the dataset creation
procedure involves re-projecting all land-cover maps in the same geographic coordinate
system, which involves significant deformation on raster data [274]. Those vector maps
are rasterised after the reprojection and aligning step.

**Alignment and reprojection** All maps are cropped and aligned according to France's borders and are re-projected to the French official projection system EPSG:2154. This step involves nearest neighbour resampling for maps only available in raster format to preserve the original resolution. This step produces a spatial shift for raster maps with a degradation of the geometric resolution that can reach half the size of a pixel in the input image x or y directions [260] and $\frac{\sqrt{2}}{2}$ pixels in the diagonal directions [274].

**Optional reclassification** Land-cover classes with unclear (such as *Other*) or mixed content can be reclassified at this stage by either merging them with other classes or labelling them as no-data.

**Rasterization** Vector maps are rasterised. The resolution is determined using the official target pixel resolution when provided or is arbitrarily considered as half the MMU value.

**Patch decomposition** CNN are originally built to analyse traditional photography, which exhibits a limited number of pixels (often less than 1, 000 × 1, 000) or can be resized. Conversely, this assumption does not hold for the obtained raster land-cover map. The number of pixels is often very high (120, 000×120, 000 for a ten-meter map covering all of France), and can not fit directly in memory when processed by a CNN. A common strategy is to rely on a grid-based approach to decompose the wide image into a subset of smaller images, as represented in Figure 3.1. Each of these small images, termed *patch*, is then processed by the CNN like it would have been for traditional photography. One of the main limitations of the patch-based approach is that areas close to the edge of the patch only have access to a limited amount of spatial information, as up to half of it might be outside of the patch (Figure 3.1). To circumvent this issue, a common strategy is to process spatially overlapping patches. However, this strategy is incompatible with some of the random train/test split procedure presented in the following paragraph as it would make it possible to identify neighbour train/test patches using the overlapping area. Of course, in an operational setup in which one wants to produce a France wide map, one should use spatially overlapping patches. However, for experimental purpose, experiments are carried out with non overlapping patches to enable unbiased measurements. We choose to perform translation on a 6×6 km² which ensures reasonable memory consumption (600×600 pixel for a 10m land-cover map and 60×60 for a 100m one), while limiting the proportion of pixel under edge influence.
**Train/test/validation split** Patches are separated into a train (65%), test (35%) and validation (5%) to ensure that the CNN is not evaluated on the data used for training it. This separation can either be random or geographically organised, as illustrated in Figure 3.2. The first strategy enables estimating accurate metrics over France. In contrast, the second is more appropriate for evaluating spatial generalisation ability and is more difficult to analyse as it highly depends on the chosen geographical repartition. We choose to rely on random sampling to design this dataset to simplify the analysis of the results and make the quantitative translation result comparable to those assessed by the original land-cover producer.

<figure><img src="image_15.png" /><figcaption>Figure 3.1: Patch based approach. A France wide map is split into small tiles to enable processing by a neural network. This strategy creates a detrimental loss of information on the edge of the patch as each patch is analysed independently from others (right). For visibility issues 20×20 km² patches are represented while 6×6 km² are used.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 3.2: Separation of train (green), test (purple) and validation (blue) data performed randomly (left) or with a spatially systematic approach (right).</figcaption></figure>
## 3.2 Study area and used land-cover maps

This section focuses on the land-cover maps used for the experimentation presented in this PhD manuscript. The maps shown here were selected from the numerous existing ones based on four criteria.

1. **Spatial extent:** To ensure spatial overlap between maps, we selected a spatial extent on which our study is conducted: the whole 550,000 km² of mainland France territory. Selected maps should be broad enough to ensure enough class and landscape diversity. We arbitrarily set up a 1% overlap with the territory selection limit. We underline that the smaller the spatial extent is, the less geographical context there is. Learning to analyse geographical context on a small extent land-cover map, even though possible, is less interesting as it provides results close to a non-context wise method.

2. **Nomenclature diversity:** Selected land-cover maps should at least include ten classes to ensure a minimum diversity of classes. Otherwise, the translation is either straightforward (as all the pixels of the source class should only be translated into a single target) or impossible (when the source class shall be translated into multiple target as the less source classes the less spatial context).

3. **Semantic accuracy:** Selected maps should exhibit at least a 70% overall accuracy to avoid working on too noisy data. The 70% bound is chosen arbitrarily to limit the number of classes with a per-class accuracy below 50%. Indeed each class with an accuracy below this number holds more erroneous than reliable examples and is deemed to enforce the model to learn erroneous translations. We underline that removing all land-cover maps holding at least one class below 50% accuracy per class accuracy is, in practice, unfeasible as it would remove too many automatically classified maps.

4. **Open access license:** To be selected, a land-cover map must be published under an open access policy. This ensures that datasets can be shared to foster more research on land-cover map translation.

This first section gives some insightful details about France's territory's characteristics. We then review the characteristics of the six-land-cover maps fulfilling the previously mentioned criteria. Lastly, we provide a first analysis of the associated challenges in terms of compatibility for semantic translation and machine-learning-related challenges.

### 3.2.1 Study area

Metropolitan France, i.e. the part located on the European continent, covers an area of 550,000 km², making it the third largest country in Europe. This vast spatial extent enables building a large patch-based dataset, ensuring sufficient data to perform train/test
splits between patches with all classes and features in both datasets. Moreover, it avoids
the need to restrain the patch size to very small spatial extents like the 32×32 pixels
of [273], which necessarily limits the spatial context available.

France's varied landscapes make it challenging for land-cover map translation. As a coastal country, it is bordered by maritime facades to the North, West and South, with a total length of coast greater than 3,427 km. These coastlines offer varied landscapes, from fallout mountain ranges in the South-East, plateaus ending on cliffs in the North or vast sandy plains in the South-West. France's territory also offers various topographical units via multiple eroded massifs (Armorican, Central, Corsica ...) or higher ones (Alpine, Pyrenean) with a peak at 4,808 meters above sea level. These massifs delimit several sedimentary basins, especially the Aquitaine Basin to the South-West and the Paris Basin to the North. The latter exhibits particularly fertile soil resulting in an agricultural land spatial distribution asymmetry.

<figure><img src="image_17.png" /><figcaption>Figure 3.3: Eco-Climatic areas defined by [154]. For convenience, areas under 100km² are removed. We added a distinct Ecoclimatic area for Corsica, which was not represented in the original map (this climate is close to the Mediterranean one).</figcaption></figure>

The fairly marked climate regional variations of metropolitan France are also interesting
as they enforce intra-class diversity and spatial contexts, *i.e.* forest tree density or tree
essence vary significantly depending on the climate. France's climate is organised in two
gradients. The Atlantic Ocean on the West coast creates a West-East gradient with a
higher wind exposure and annually more stable temperature on the West. Additionally,
the traditional North-South gradient, with higher temperatures in the South (Northern hemisphere), is reinforced by the presence of the Mediterranean sea in the South. These climates can all be declined in more local variants due to mountainous areas (East and West borderline). Figure 3.3, for instance, presents an Eco-climatic area delineation into 9 areas proposed by [154].

France also encompasses broadly different land-cover types heterogeneously distributed. About 5% of its territory is currently artificialised in scattered urban blocks, the Paris region alone representing nearly 20% of the population while accounting only for 2% of the country's total surface. This asymmetry is materialised by a "diagonal of low densities" from North-East to South-West, characterised by a low population compared to the rest of the country. Most of the surfaces are dedicated to agriculture (60%), with a prominent contrast depending on the region's climate and geological characteristics, e.g. a fragmented hedgerow landscape on the West side of the country and Open-field areas in the Paris region. France is also one of the most wooded countries in Western Europe, with forests occupying 34% of the metropolitan territory, mainly in the form of deciduous trees but also of conifers in the mountains and the planted forest of the moors to the West. Wetlands, which covered nearly a quarter of the country, have declined sharply since the 19th century and represent less than 1% of the territory to date, as do lakes and rivers.

To sum up, the broad range of landscapes and climate makes the study area interesting for evaluating land-cover translation in a wide variety of situations. However, the prominence of two super-class, Forest and Agricultural land (94% of the territory), enforces the selection of maps with rich nomenclatures. For instance, translation into a simple 5-class nomenclature, including a Forest and a Cropland class, easily achieves high-quality results.

### 3.2.2 Selection of a restricted number of land-cover maps

Numerous land-cover maps respect the four criteria presented above. For instance, amongst the land-cover maps reviewed by [106], six global (MCD12Q1, GlobCover, GLCNMO, GLC SHARE, GeoWiki) and 3 Europe comprehensive (CORINE, LUCAS on 2km grid, GlobCorine) land-cover products respect the above criteria. Additionally, other global products proposed since 2015 (GlobLand30, ESRI LULC) and various France-wide (OSO) or local scale land-cover maps (OCSGE, MOS with one map per administrative area, CRIGE-PACA) all match previous criteria. As we can not analyse all possible translations, we arbitrarily choose to focus only on six maps. The selection of those six maps tries to maximise the complexity of translation by enforcing high diversity in production method, spatial resolution, nomenclature and spatial extent.

Amongst those four characteristics, the production method is the most important to diversify as it influences the three other ones. Photo-interpreted land-cover maps usually exhibit rich nomenclatures (more than 15 classes), often mixing land-use and land-cover definitions, coarse spatial resolution and are often coarser when they cover a wide spatial extent. Unlike most automatically obtained land-cover maps, they exhibit a minimum
mapping unit which correlates the classification of one pixel to its surroundings. They often exhibit fewer errors (around 10%) than automatically derived ones, with most errors being region shaped, i.e. multiple contiguous pixels are all misclassified identically and with the same magnitude per-class error ratio. Conversely, most automatically derived land-cover maps exhibit smaller nomenclatures (less than 15 classes), mainly land-cover oriented, as land-use is often challenging to obtain from images. Their spatial resolution is often significantly higher, and they rarely exhibit a minimum mapping unit, i.e. the content of one pixel is less dependent on its surrounding. They often exhibit significantly higher error rates (up to 35%), most errors exhibiting systematic spatial (edges between land-cover) and class (class close in the feature space are more likely confused) patterns. Nomenclature, resolution and error patterns of automatically obtained maps highly depend on the source data used (MODIS, Sentinel, PROBA-V...) and the algorithm characteristics (pixel classification vs OBIA classification, RF vs SVM vs Deep learning).

Lastly, the selected land-cover maps should be temporally close to ensure that the learnt translation does not suffer significantly from a temporal gap between the source and target land-cover. We arbitrarily set this closeness limit to 10 years.

### 3.2.3 Presentation of the input land-cover maps

Table 3.1 summarises the main characteristics of the six land-cover maps selected which exhibits a broad range of production methods (either photo-interpreted or automatically generated, different Source data), spatial resolutions (from 10 to 100 m), nomenclatures (from 11 to 44 classes, cover and use) and spatial extent (from 10,000 to 550,000 km²): CGLS-LC100 [33], CORINE land-cover map [218], OSO [139], OCS-GE cover, OCS-GE use, and MOS.

<table><caption>Table 3.1: Main characteristics of the six selected LULC maps.</caption><thead><tr><th></th><th>CGLS-LC100 [33]</th><th>CLC [218]</th><th>OSO [139]</th><th>OCS-GE cover [229]</th><th>OCS-GE use [229]</th><th>MOS</th></tr></thead><tbody><tr><td>Extent</td><td>World</td><td>Europe</td><td>France</td><td>West and South France</td><td>West and South France</td><td>Paris area</td></tr><tr><td>Generation</td><td>Machine-Learning</td><td>Photo-interpreted</td><td>Machine-Learning</td><td>Photo-interpreted</td><td>Photo-interpreted</td><td>Photo-interpreted</td></tr><tr><td>Source data</td><td>PROBA-V</td><td>Landsat, Sentinel-2</td><td>Sentinel-2</td><td>Aerial imagery</td><td>Aerial imagery</td><td>Aerial imagery</td></tr><tr><td>Distribution format</td><td> raster</td><td>vector</td><td> raster</td><td>vector</td><td>vector</td><td>vector</td></tr><tr><td>Selected year</td><td>2018</td><td>2018</td><td>2018</td><td>2014-2015</td><td>2014-2015</td><td>2017</td></tr><tr><td>Number of classes</td><td>12</td><td>44</td><td>23</td><td>14</td><td>17</td><td>11</td></tr><tr><td>Pixel resolution (m)</td><td>100</td><td>100</td><td>10</td><td>10</td><td>10</td><td>20</td></tr><tr><td>Minimum mapping unit (m²)</td><td>10,000</td><td>250,000</td><td>100</td><td>200-2,500</td><td>200-2,500</td><td>400</td></tr><tr><td>Official geometric accuracy (m)</td><td>100</td><td>100</td><td>10</td><td>5</td><td>5</td><td>5</td></tr><tr><td>Official semantic accuracy</td><td>73 % (Europe)</td><td>92 % (Europe)</td><td>87 % (France)</td><td></td><td></td><td></td></tr><tr><td>Accuracy on ground truth</td><td>80 % (France)</td><td>88 % (France)</td><td>86 % (France)</td><td></td><td></td><td></td></tr></tbody></table>

In order to reduce the impact of changes occurring between two maps in the translation procedure, we carefully selected the year of the maps to make them the closest possible to each other (selected years are indicated in Table 3.1). Few maps are produced on a yearly basis which inevitably generates discrepancies between the six maps.

The Copernicus Global Land Service land-cover map (CGLS-LC100) map has global coverage and is released annually in raster format. Based on PROBA-V image time series
classification with a supervised Random Forest framework [33], each map covers a civil
year reference period with five released versions so far (2015, 2016, 2017, 2018, 2019) <fnref n="1" />.
Main map characteristics include a spatial resolution of 100 m, up to 22 classes (with a
fine-grained separation into 12 forest labels), and hierarchically organised into a 3-level
nomenclature. Level 1 merges all forest classes into one (leading to 11 classes), and level 2
distinguishes open from closed forests. We choose to rely on the level 2 nomenclature (see
Table A.6), instead of the level 3 due to its higher accuracy (estimated overall accuracy
over Europe of 80% at level 1, 73% at level 2 and not communicated at level 3 [310]).
Indeed, our proposed solution relies on a supervised learning process: inserting a too
significant noise level would be detrimental [224]. Moreover, working with level 3 labels
would have also required dealing with complex classes such as *Unknown open forest types*
that are poorly handled by translation systems.

<figure><img src="image_18.png" /><figcaption>Figure 3.4: Spatial extent of the 6 land-cover maps used in this work.</figcaption></figure>

The CORINE land-cover (CLC) database and its 92% thematic accuracy [218] has been
the reference for land-use and land-cover map documentation at the European scale for the
last three decades. As part of the European project Copernicus, five versions of the product
have been released so far (1990, 2000, 2006, 2012, 2018)<fnref n="2" />, covering up to 39 countries in
2018. CLC is mainly generated through visual inspection of both mono and multi-temporal
high resolution/very high-resolution optical satellite images (Landsat, Sentinel-2, SPOT),

<footnote n="1">¹https://land.copernicus.eu/global/products/lc</footnote>

<footnote n="2">²https://land.copernicus.eu/pan-european/corine-land-cover</footnote>
complemented with local databases. CLC is released dually in vector format with a 250,000 m² minimum mapping unit (MMU) for classes represented by polygonal objects and an additional 100 m width constraint for linear features and in raster format with a 100 × 100 m pixel spatial resolution. The nomenclature includes up to 44 classes (see Table A.2), hierarchically organised into a 3-level nomenclature. Most current translation methods rely on the first or second level of nomenclature as the translation accuracy highly depends on the semantic and spatial correspondences between the source and the desired nomenclatures, which increase for land-cover map with few classes [20]. Conversely, in the following, we target full CLC level 3 translation (44 classes) to understand better and assess which classes can be distinguished using contextual methods. Indeed, context-based translation solutions exhibit a significant potential for some challenging CLC level 3 classes (e.g., *Mixed Forest*, or *Green urban areas*) that calls for fine assessment.

The Occupation des Sols Opérationnelle (OSO) covers Metropolitan France and is released annually in raster format. Based on Sentinel-2 image time series classification with a supervised Random Forest framework [139], each map covers a civil year reference period with six released versions so far (2016, 2017, 2018, 2019, 2020,2021). Main map characteristics include a spatial resolution of 10 m, 23 classes with a fine-grained 11 class agricultural discrimination (see Table A.1), and an overall accuracy higher than 85%. This product is valuable to this study for its high resolution coupled with a detailed crop nomenclature. The OSO product is freely distributed around April each year <fnref n="3" />.

The Occupation des Sols à Grande Echelle (OCS-GE) map covers West and South-West France (125,000 km²), and is expected to be updated at least on a 5-year basis. Based on photo-interpretation of aerial visible and near-infrared imagery at 20 cm, each administrative state is mapped independently, with the first campaign between 2014-2015 and one between 2019-2021(⁴). Our work only includes 2014-2015 maps, the more recent one still being under review at the moment of this writing. Main map characteristics include a spatial class-dependent resolution between 5 and 10 m, a minimum mapping unit between 200 and 2,500 m² depending on the class and the location and two land-cover map/land-user nomenclatures: 14 labels for land-cover map (see Table A.4) and 17 for land-use. This joint LC/LU product is particularly interesting in studying automatic land-use prediction from land-cover map. So far, the two products are generated on the same spatial support: a territory segmentation is automatically performed using a database with road and rail network to obtain a global skeleton which is later subdivided in more refined units by photo-interpreters independently for land-use and land cover. In the remainder, we refer to those two nomenclatures as **OCS-GEc** for land-cover map and **OCS-GEu** for land-use. The choice has been made to remove the following three classes from OCS-GEu: *Other primary productions*, *Other transport networks* and *Unknown use*, due to their mixed and complex content.

The Mode d'Occupation des Sols (MOS) map covers the Paris region (12,000 km²) and is

<footnote n="3">³https://www.theia-land.fr/en/product/land-cover-map/</footnote>

<footnote n="4">⁴https://geoservices.ign.fr/ocsge</footnote>
released approximately every four years in vector format. Based on the visual interpretation
of 0.15 m aerial optical imagery, each map covers a civil year reference period with nine
released versions so far (1982, 1987, 1990, 1994, 1999, 2003, 2008, 2012, 2017)<fnref n="5" />). Main
map characteristics include a spatial resolution of around 20 m, up to 81 classes (with a
fine-grained 68 built-up classes), hierarchically organised into a 4-level nomenclature. The
choice to rely only on the 11 class level 1 nomenclature (see Table A.3) has been made
since the other levels are not freely available.

## 3.3 MLULC challenging characteristics

This section briefly highlights some of the main challenges faced when translating between
the six selected land-covers. We identify four principal challenges: nomenclature translation,
resolution translation, translation of erroneous land-cover maps, and spatial generalization.

### 3.3.1 Nomenclature translation issues

#### 3.3.1.1 Quantifying the nomenclature translation issues

We provide in Appendix B various tables describing independently for each source/target
map the main semantic correspondences from one class to another based on a manual
analysis of the main semantic links (see Section 6.2.2.4 for more detail on this procedure). In
particular, we established that amongst the independent translation of the 118 accumulated
classes of the dataset into each of the five other nomenclatures (590 possible source/target
association), only 315 (53.3%) source classes have a single relatively close correspondence
in the target nomenclature (denoted 1-to-1 translation), i.e. for which one translation
appears more consistent than all the others. The translation of those 315 source classes
results in 151 target classes; thus 25.6% of classes are semantically easily obtainable.
Conversely, 11.6% of classes can never be obtained by semantic translation of a source
map due to a lack of semantic correspondence (denoted 0-to-1 translation). Appendix C
presents for each source to target map translation the proportion of target classes obtained
from a 1-to-1 and 0-to-one translation (with two different techniques) to help identify
the most challenging translation scenarios from a semantic point of view. The main
observation is that the only source map usable to obtain more than 50% of target classes
using a pure 1-to-1 semantic approach is CLC due to its high number of classes. All
other source-to-target translation scenarios exhibit configurations in which the proportion
of target classes obtained by translating one source class into multiple targets is above
70%. Obtaining high-quality semantic translation requires defining context or ancillary
data-based criteria to translate each of those source classes into their multiple possible
translations. Synthetically, the most challenging class to obtain are OSO fine grained

<footnote n="5">⁵https://www.data.gouv.fr/fr/datasets/mode-doccupation-du-sol-mos-en-11-postes-en-2017/</footnote>
11 agricultural classes, CLC 11 urban classes and 5 wetland oriented classes and almost all OCGSE use classes due to a lack of semantic correspondences in the other land-cover maps.

### 3.3.1.2 Imperfect semantic matching

We underline that even source classes with a clear principal semantic correspondent in the target nomenclature (1-to-1) often corresponds to an imperfect translation due to: variations in class characteristics, temporal patterns and arbitrary rules.

**Varying class characteristics** Class definition often includes a threshold on specific characteristics expected for an object to be attributed to the corresponding classes. Those thresholds are often widely different depending on the considered land-cover map. For instance, we provide below the threshold definition to attribute the class *Broad leaved forest* for CLC (2), OSO (3), and OCSGE cover (4).

"The predominant classifying parameter for this class is a crown cover density of > 30 % or a minimum 500 subjects/ha density [...] The minimum tree height is 5 m." - CLC (2)

"Land with an absolute tree cover rate greater than or equal to 10 % with trees reaching or capable of reaching a height greater than 5 meters on-site" - OSO (3)

"Absolute tree cover rate greater than or equal to 25%" - OCSGE cover (4)

Even though translating one of those classes into one of the other is the natural translation, it results in inaccurate translation for objects between the source and target thresholds. As many such inconsistencies are observed in the dataset, we can not present them all. However, we underline that this class characteristic variation mainly affects biotic land-cover types.

**Varying temporalities** We highlight that due to the production scheme of CLC, CLC class definition tends to exhibit a multi-year temporal pattern, especially for agricultural areas, shrubs and forest areas (see Section 2.3.3.1). Conversely, all other maps tend to define their classes on a year-based analysis, making the translation to CLC difficult when only one source map is used. The translation from CLC to one of those maps is ill-defined in those classes if no additional data with a date close to the target map is used.

**Arbitrary rules** Land-cover maps often include sets of arbitrary rules, mainly in forms of spatial constraints that separate different land-cover types. Most affected translation includes the distinction between different sorts of water objects, such as the separation between salted and non-salted waters in Figure 3.5 (i.e. CLC *Water courses*, *Water bodies*, *Coastal lagoons*, *Estuaries*, *Sea and Ocean*, *CGLS*, *Permanent waters bodies*, *Ocean*). Occasionally some other land-cover types can exhibit those spatially arbitrary delineations. For instance, OCSGE use *Secondary or tertiary production and residential usage* seems
to include all elements inside an arbitrary, non-constant across-territory buffer around built elements while not giving information on this buffer in the class definition. Therefore translating those classes implies to infer those arbitrary rules directly from the map by using additional data or inferring them from context.

<figure><img src="image_19.png" /><figcaption>Figure 3.5: Illustration of the resolution and semantic discrepancies between OSO and CLC. See Tables A.2 and A.1 for the detailed nomenclatures.</figcaption></figure>

### 3.3.1.3 Constraints specific to machine-learned translation

The high over-representation of forest and agricultural lands in France (94%, see Section 3.2) compared to the rest of land cover fosters a significant inter-class imbalance, especially for maps with a high number of urban or wetland-oriented classes such as CLC and MOS or land-use oriented such as OCSGEu. For instance, in the CLC map, the ratio between the most and the least represented classes is higher than 200 in level 2 and higher than 7000 in level 3. When performing manual semantic translation, this does not represent a significant issue as long as the statistically most frequent classes are advantaged compared to statistically less observed one on 1-to-n translations (which is not the case in most current translation frameworks). However, from a machine-learning point of view, this is challenging as it requires a solution that is not too favourably biased towards the most statistically probable translation.

## 3.3.2 Resolution translation issues

In the dataset, the gap of pixel resolution is of a factor of 100 between the most resolved maps (OSO, OCSGE) and the least resolved ones (CGLS-LC100, CLC) and goes up to 2500 when the minimum mapping unit is taken into account. Figure 3.5 illustrates this resolution gap between OSO and CLC, i.e. 100 OSO pixels are resumed into one CLC pixel, and each CLC segment at least includes 25 pixels (one segment resume at least 2500 OSO pixels).