the influence of neighbouring pixel classes. This method, called A-Unet, is based on
adapting a convolution network to enable it to translate the resolution and nomenclature
simultaneously. The analysis of the results highlights several important points. First,
the joint use of the shape and the neighbourhood significantly improves the quality of
the translation both visually and qualitatively: +5% OA<sub>ag</sub> and +8% mF1<sub>ag</sub> on average
compared to the best baselines. However, we point out several limitations of the method.
It is generally less good at preserving translation geometry than baselines and does not
generalise well spatially. Furthermore, we observe that the map with the smallest spatial
extent (MOS) does not benefit significantly from using context. We explain this mainly
by the small size of the training set used. Observing that a change of architecture does
not improve the results, we postulate that efforts should focus on adapting the training
procedure.

The following section focuses on the geographical context. We start with the hypothesis that using geographic coordinates is a simple way to learn this context based on the observation that two close coordinates share the same geographical context. Experiments show that this method is more likely to improve the translation than considering a predefined ecoclimatic zone. Based on this observation, we propose incorporating the geographical coordinates into the network. Making the same observation as other works, we show that integrating the coordinates directly into a network does not work. We propose an approach based on positional encoding, a strategy that has proven itself in NLP. The results demonstrate the method's effectiveness, even when the size of the training set does not allow fine learning of the geographical context. In particular, we demonstrate that the proposed approach is always superior to coordinate-free translation.

The fourth section focuses on the temporal context of translation from two different angles. We first question the relevance of using multiple dates from a source map to improve the translation quality. We start from the observation that multiple classes include a notion of temporality, i.e. it is class A if class A has at least been observed once in the last three years. We hypothesise that the temporal order of the source maps does not matter because the definitions never precisely stipulate a date but always a period. We show that the incorporation of this temporality allows a significant increase in the quality of the translation, mainly for the target classes, including a notion of temporality, but also, in a lesser way, for others. We explain this by the fact that combining several source maps allows the method to distinguish errors in the source data better improving the translation quality. Secondly, we emphasise the existence of a temporal context specific to the "learnt" translation paradigm: the temporal spacing between the source and target data used to learn. We hypothesise that the temporal spacing between the source and target maps adds only little noise to the learning compared to the noise generated by the errors in the original maps, as long as the learning is carried out on a large area and that the time spacing remains reasonable. We propose three distinct learning scenarios corresponding to different operational situations. Results show that the temporal spacing only very slightly deteriorates the results. In particular, there is no significant difference between training a
network to translate two maps with the same date (scenario 1) and two maps with a time
gap (scenario 3). In both cases, the resulting map gives a translation corresponding to the
date of the source map. This critical finding guarantees the operational usability of learnt
translation methods.

Lastly, we focused on tackling the cartographic context of translation, which involves
working on, by nature, noisy data. We first underlined that creating a source noise resilient
translation method is a complicated task as it involves evaluating instance-specific noise
probabilities. We argued that the detrimental impact of source noise on translation was
limited as errors tend to exchange two semantically close classes that tend to have the
same translation. Moreover, as our spatial and temporal context-aware methods can
already partially evaluate those source noise characteristics, we believe that efforts must
concentrate on tackling the target noise effect. We presented a method for tackling target
noise based on the idea that the land-cover translation exhibits specific characteristics:
a large amount of annotation at each iteration and an available noise transition matrix.
Thus we propose to compute the divergence between the expected per class distribution
(noise transition matrix) and the observed one. Despite being unable to assess statistical
significance due to ground truth size, our method seems to outperform commonly used
loss-based correction strategies in terms of output class diversity (+7% mF1gt compared
to cross-entropy). Furthermore, the methods appear to be resilient to a significant noise
level in the noise transition matrix used for training, making it usable in real operational
scenarios. We believe additional experiments should be conducted on this loss function
using a broader ground truth and more datasets to comfort the results.

As the experiments were carried out on a wide variety of translations (up to 26 different),
we are confident in the ability to generalise these results to other land-cover maps than
those in the data set. By focusing only on these 26 translations, we can identify some
limitations of the proposed methods. In particular, the use of context to translate CLC
to any other map generally offers only slight improvement, which is explained by the
fact that the difference in resolution is considerable with the other maps (up to a factor
of 2500) due to CLC's MMU. Therefore, it seems essential to propose data insertion
methods that are more spatially resolved to improve the translation quality. This would
also improve the preservation of target geometry which, as evidenced by the value of EPI,
is poorly preserved by the methods presented. This would also significantly increase the
values of OAgt and mF1gt, which remain, with the notable exception of the OSO to CGLS
translation, lower than that of the original products. Lastly, the methods suffer from a
significant lack of generalisation ability which should be compensated for applications
such as extending a land-cover map on a broader spatial extent using a source map with a
broad spatial extent.
# Multi-land-cover, Multi-modal translation

## 5.1 Multi-land-cover map translation

### 5.1.1 Motivation

In the previous chapter, we introduced a Convolutional Neural Network based encoding-decoding strategy to translate land-cover maps using spatial context. The core idea lies in the possibility for each map pixel to be translated differently depending on its close surrounding pixels. However, this supervised method is designed to perform mono-land-cover map translation, i.e. a single source is translated in a single target. It requires the two maps to overlap, at least partially, spatially. When impossible, a pivotal map that spatially overlaps the two others might be used, but this is likely to drastically lessen the translation performance by requiring two translations instead of a single one. Multiple translations require separate training phases. With few training samples, the performances are likely to be limited.

Recently, deep learning methods have achieved state-of-the-art results in natural language processing and, more precisely, in language translation [59, 307]. State-of-the-art methods have shown the superiority of multi-lingual trained models against their mono-lingual counterparts [59], especially for languages with a small number of translation examples. Multi-language training seems to benefit from the obtained multi-language common representation space [245]. Finding shared representations is also frequently addressed by the remote sensing community for combining multi-modal data from various sensors, with varying resolutions and information into a compact and discriminative embedding [11, 127, 128, 223]. Surprisingly, this question remains unaddressed for the land-cover map translation task at a pixel or object level. Therefore, this section tries to answer the following question: can we find a shared space for multi-land-cover map translation that would be beneficial for their individual generation?

Even though a current trend in computer vision is to address the projection into common representation space with vision transformers, we adopted a Convolutional Neural Network-based solution that requires far less training and is thus suitable even for land-cover with
small spatial extent (Figure 5.1). Independent U-Nets are trained to project each land-
cover map into a shared representation space. This, tailored for translation representation
space, enables both translation and the self-reconstruction of the input land-cover maps
to ensure that no information is lost during the mapping to the shared space.

<figure><img src="image_45.png" /><figcaption>Figure 5.1: Overall multi-land-cover map translation architecture. Our network (blue boxes) is trained to perform both self-reconstruction and translation. There is no restriction in the number of maps that can be embedded into the common representation. For convenience, we only represent two maps (A and B). Red and orange arrows represent the possible paths for maps A and B, e.g. A can be either translated into B or self reconstructed into A. At inference, only one source map is required.</figcaption></figure>

The key contributions of this section are summarised as follows:

* We propose a method to perform multiple translations at the same time using a single translation model.

* We ensure that the method projects all land-cover maps into a shared representation space in which two different maps of the same extent are represented closely. This ensures that the shared representation space encodes elements with comparable characteristics in terms of semantics and context independently from the land-cover map used, increasing spatial generalisation for maps with limited spatial extents.

* We conduct a comparative evaluation of the approach with the mono-land-cover map translation presented in the previous chapter.
## 5.1.2 Method

### 5.1.2.1 Training protocol

We aim to find a simultaneous transformation of the nomenclature and spatial resolution of the six maps (see Section 3.2). Inspired by the literature review conducted in Section 2.4, we enforce the translation to use an intermediate common representation space for all maps. This representation in a common space is referred as an "embedding". This leads to reach two consecutive objectives: 1) project each map into a shared embedding space; 2) decode this embedding into each of the six maps.

Inspired by recent works on multi-modal data representation [39, 133, 152, 337], we propose to train separate encoders and decoders for each map, and subsequently use cross-reconstruction to enforce common representations of land-cover maps representing the same spatial extent (see Figure 5.2). We train our network to both reconstruct a given land-cover map with one decoder and to translate into the desired target land-cover map with another decoder. This dual objective enforces the embedding to be rich enough to preserve all source map information (reconstruction) while encoding it suitably for translation.

Even though cross-reconstruction encourages the learnt embedding to be comparable for all land-cover maps, it does not guarantee it. Therefore, many of the previously cited works also included a constraint on embedding pairs of corresponding data (e.g., using adversarial training or a loss term for embedding comparison). We adopt the latter strategy which avoids the convergence complexity of adversarial training. A Mean Square Error between embeddings covering the same spatial extent is thus computed.

Instead of computing the loss for all available maps covering one spatial extent, the network is trained by computing the loss for only one pair of maps at each optimisation step. This pair-wise optimisation is used as a workaround for GPU memory limitations. Indeed, land-cover map translation requires large image patches (600×600 pixels × the number of classes) to account for the MMU of some of the maps. Also, simultaneously training multiple networks is memory consuming. This pair-wise optimisation enables the use of a larger batch size and achieves a better result than optimising all different maps simultaneously on smaller batches. This iterative pair-wise approach is also the one generally used in multi-lingual model training [59].

To sum up, at each iteration, two patches, A and B belonging to different maps but representing the same spatial extent are encoded by distinct U-Net producing Encoded A and Encoded B. A first loss term $L_{emb}$ (detailed in Section 5.1.2.3) is computed between the two embeddings to estimate their resemblance, then each embeddings is processed by each decoder resulting in two translations (Encoded A into the decoder of B, and Encoded B into the decoder of A) and two self reconstructions (Encoded A into the decoder of A, and Encoded B into the decoder of B). Two loss terms evaluating the quality of the two translations $L_{tra}$ and the two reconstructions $L_{rec}$ are then computed.
5.1.2.2 Network

The design of our Multiple Land-Cover Translation Network (MLCT-Net) is made according
to the following observations:

1. The encoder must have a sufficient receptive field to encode each object using its surroundings. Thus, the architecture is constrained by the MMU of each map. Since CLC has a 250,000 m² MMU, the theoretical receptive field should at least have a 250,000 m² width. An embedding with a ground resolution of 10 m leads to at least a 250 pixels wide receptive field. Achieving this size of receptive field using only convolution is unfeasible as it would require very large networks and would be inefficient due to gradient vanishing problem. Pooling strategies appear well suited to increase the receptieve field as land-cover often offers wide homogeneous regions limiting the information loss.

2. The decoder should remain as simple as possible to ensure that the learnt embedding remains as identical as possible for all land-cover maps. Decoders with high capacity may lead to a latent space with small information content [47].

<figure><img src="image_46.png" /><figcaption>Figure 5.2: The proposed cross-encoder architecture. In purple and green, two land-cover maps with respectively $c_1$ and $c_2$ classes and $r_1 \times r_1$ and $r_2 \times r_2$ pixels. We represent in orange the common embedding space.</figcaption></figure>

We develop the architecture illustrated in Figure 5.2. It is mainly composed, for each map, of a (1) a nearest neighbour resampling to the highest spatial resolution (10 m), (2) a U-NET [257] encoder, (3) and a pyramidal spatial pooling [44] followed by a 1-pixel wide kernel convolution layer as a decoder. This architecture meets each of the above criteria.
The resampling enables using the same architecture for each map. However, this strategy is only feasible if the gap between the lowest and the highest resolutions remains reasonable compared to the GPU memory. A low resolution enforces patches to cover a wide area to get a grasp of the spatial context. This results in very large patches for the maps with higher resolutions. The U-Net addresses the receptive field size by down-sampling the input multiple times, which is more memory efficient than increasing the network depth. Our encoder architecture exhibits only two differences with the original U-Net architecture. The first one is the use of Group Normalisation [332] instead of Batch Normalisation. This enables a stable normalisation, even on a small batch size. The second one is the use of 5 down-sampling blocks, instead of 4 in the original paper, to widen the receptive field.

### 5.1.2.3 Losses

At each optimiser step, the loss is computed for one pair of maps using Equation 5.1:

$$L = L_{rec} + L_{tra} + L_{emb}. \qquad (5.1)$$

Let $E_A(A)$ denote the result of encoding $A$ with its dedicated encoder and $D_A(E_A(A))$ denote the processing of $A$ by its dedicated encoder-decoder (self reconstruction). $D_A(E_B(B))$ denotes the encoding of $B$ is processed through the decoder $A$ (translates $B$ into $A$). We denote $E_A^i(A)$ the $i^{th}$ dimension out of $w$ dimensions of the encoded version of $A$.

$L_{rec} = L_{CE}(D_A(E_A(A)), A) + L_{CE}(D_B(E_B(B)), B)$ is the reconstruction loss enforcing the embedding to preserve map-specific information: computed as the sum of cross-entropies ($L_{CE}$ see Equation 4.3) between the two self-reconstructed and their respective sources.

$L_{tra} = L_{CE}(D_A(E_B(B)), A) + L_{CE}(D_B(E_A(A)), B)$ evaluates translation quality: computed as the sum of the two cross-entropies of the two translated maps and their respective targets.

$L_{emb} = \frac{1}{n} \sum_{i=1}^{w} (E_A^i(A) - E_B^i(B))^2$ is the Mean Square Error (MSE) loss between the embedding of the two source maps which enforces the representation to be shared between maps. The global loss is theoretically minimal when the three following assumptions are simultaneously met: 1) the self-reconstruction of each map is perfect; 2) the translation is also perfect; 3) embeddings on the same areas are identical.

The Cross entropy combined with Dice loss explored in the previous chapter can be used here instead of traditional cross entropy to improve mF1. However it complexifies the loss significantly by adding four more terms making the optimisation more challenging. As the optimisation take more times, using such loss would have reduced the number of experiment presented in this section with this combined loss all the experiments presented in this section are carried using solely the cross entropy.
#### 5.1.2.4 Geographical context

The same geographical coordinate encoding strategy as in the previous chapter is used. However, we add a softmax transformation after the MLP and replace the addition by a multiplication between the geographical context and the embedding of each map (Figure 5.2). Softmax followed by a multiplication aims to maintain the generalisation ability to areas unseen during training.

Since each translation does not necessarily need the same geographical context information, one could learn one geographical context per pairwise translation. However, it would be impossible to generalise the translation to an area of the target map unseen during training. For example, learning a specific geographical context for the OSO-to-MOS translation is only possible on the Paris region. To preserve the common representation space, we train a unique MLP on the set of coordinates of the patches. This unique geographical context representation slightly worsens the translation quality compared to learning a per translation representation. However, it is the only way to maintain the ability to translate maps on areas not included in their original spatial extent.

#### 5.1.2.5 Comparisons

Since, to the best of our knowledge, no other multi-land-cover map translation method has been published, we compare our approach to the methods introduced in Chapter 4: the baselines and the 26 A-UNet trained independently. We refer to those multiple A-UNet as the mono-land-cover map translation method. Results with the multi-map approach are expected to be better than with the two first non-contextual translation methods. They should be at least on par with the mono-map translation method, and better on land-cover maps with few training patches, as observed in natural language processing.

### 5.1.3 Results

#### 5.1.3.1 Qualitative assessment

Figure 5.3 presents the 12 translation results obtained on a patch belonging to the Paris region in Figure 3.4. Each row corresponds to the translation of one source map into the four others available for this area. Unsurprisingly, translations from coarse to high resolution maps results in almost similar performance than associating one unique target class to all the pixels of given source. External data (such as satellite imagery) could participate in increasing the translation performances. The second observation is that our network may face some difficulties in learning the MMU of CLC (25 pixels) as shown by the small three pixel wide urban areas (in red) in Figure 5.3 (first column). Commonly, network training leads to replicate in the predictions the bias observed in the original data. The most striking example is OSO *road* class, which has a 45% recall in the original data. It is often confused with *Industrial and commercial units (ICU)*. When learning to translate
a road from a given land-cover map source map to the OSO map, the corresponding class has a high probability of being an OSO *ICU*, as illustrated with the MOS-OSO translation case in Figure 5.3 (3^th^ row, 2^nd^ column). This also increases the difficulty in quantitatively assessing the quality of the results using the target data as reference.

<figure><img src="image_47.png" /><figcaption>Figure 5.3: Translation scenarios for all land-cover map available on a 6×6 km² patch belonging to the Paris region.</figcaption></figure>

Figure 5.5 presents a set of patches selected for their representativeness of the behaviour of the multi-LC. The first observation is that the spatial context influences the translation mainly on object edges, especially when the source exhibits a low resolution, e.g. in the first row, the border of a CLC *Discontinuous urban* area is translated into an OSO *pasture* area. The second observation is that when the gap between spatial resolutions remains
limited, the translation provides a successful context-dependent translation (i.e., the same class is translated differently depending on its neighbourhood), as shown for example in Figure 5.5 (second row): OSO *sparse urban* and *Industrial and Commercial Units* are satisfactorily translated into either MOS *Individual housing*, *Collective housing* or *Activity areas*, depending of each source class density or on the third row, where MOS *Forest* is translated into CGLS-LC100 *Open forest* or *Closed forest*, depending on the elongated shape of the object. Thirdly, despite context, some cases remain difficult without external data. This could for example be used in the fifth row where an OCS-GEc *Water* area must be translated into the land-use counterpart. Most of the time, such areas are classified as *No-use* in OSC-GEu. However this water lake is used for farming which the network fails to predict. This difficult case illustrates well the main limitation of this method.

Figure 5.6 presents the qualitative comparison of Statistic, Semantic, Mono-LC, and MLCT-Net on the same spatial extents. A first observation is that mono-land-cover map and MLCT-Net gives visually almost identical results. The two methods outperform the semantic and statistic baselines when source classes have multiple probable translations. For instance, for OCS-GEu (G2) to CGLS translation, "Agriculture areas" are translated solely into "croplands" by the semantic method while being translated quite accurately both into cropland and pastures by the context-aware methods. The same observation holds for urban areas in the OCS-GEu to OCS-GEc translation (and OCS-GEc to OCS-GEu). A second observation is that pure semantic based translation outperforms other methods on erroneous classes in the original target data. For MOS to OSO translation, roads (black on the MOS map) are always translated into industrial and commercial units except by the semantic baseline. This behaviour is learnt from the original OSO map, which often presents this confusion. Conversely, the learnt methods outperform the semantic baseline when the source map is erroneous. In the reverse translation case, the erroneous industrial and commercial units (truth: roads) are correctly translated into roads in the MOS maps by all methods except the semantic one.

To assess if land-cover maps are all embedded in a shared representation space, Figure 5.4 presents the embedding of a 6×6 km² patch of the test set for five different maps. As displaying 30 dimension embedding is unfeasible, a dimension reduction technique is used to reduce to 3 dimensions for RGB visualisation. Principal Component Analysis (PCA) is used as, unlike methods such as T-SNE, it enables to re-use the same reduction model (trained using a subset of the training set) for various patches of test set, ensuring that the representation remains stable for all test patches. The first observation is that all embeddings look similar, which was expected through the double constraint of cross-reconstruction and the MSE computation between embeddings. The second is that edges have a particular behaviour in the embedding. This is particularly visible on coarse resolution maps (such as CLC) with a gradient on each object near the edges. We link this behavior to the higher uncertainty of the translation near object boundaries. The third observation is that the learnt embedding for coarse resolution maps has a blurrier aspect than high resolution ones. This is, for example, clear on the CGLS-LC100 embedding,
especially on **Built up** areas. We relate this behaviour to the relative broad semantic
content of such class in a low resolution map compared to a higher resolution one (i.e.,
a *Built up* area might simultaneously include trees, dense or sparse urban and roads).
Another observation is that close values in the embedding space for two classes often
reflect close semantic values. For instance, all artificial surfaces appear in light blue, all
forest types in light to strong red, all crops and pastures in dark blue. This closeness
might be beneficial for tasks such as zero shot learning since semantically close elements
are represented closely in the embedding space. For instance, the model is never trained
to translate CGLS *Ocean* in one of the MOS class as there is no ocean in the Paris
area. However translation is easy to perform as the CGLS *Ocean* representation in the
embedding space is closer to MOS *Water* than any other MOS class. The last observation is
that when one class of a land-cover map establishes a complex semantic relationship with
another land-cover map, it is often visible in the embedding. For example, the OSC-GE
cover class *Herbaceous vegetation* mixes cultivated areas and natural grasslands while all
other land-cover maps make a clear distinction between those two vegetation types. This
leads to distinct embeddings.

<figure><img src="image_48.png" /><figcaption>Figure 5.4: Shared embeddings (*below*) for five land-cover maps of interest (*top*). Colors result from a dimension reduction from the original 30-dimension embedding to 3 dimensions (RGB) using Principal Component Analysis.</figcaption></figure>
<figure><img src="image_49.png" /><figcaption>Figure 5.5: Benefits and limitations of multi-land-cover map translation. Each square highlights an area with meaningful spatial context (see text for more details).</figcaption></figure>
<figure><img src="image_50.png" /><figcaption>Figure 5.6: Visual comparison between the output of MLCT-Net and existing baselines.</figcaption></figure>

5.1.3.2 Quantitative assessment

All conceivable translation scenarios are tested. Table 5.1 reports the agreement metrics.
We remind the reader the agreement can only be computed on the intersection of the
source and the target extents. Context-aware translation methods have higher agreements
than their semantic and statistical counterparts. The improvement between contextual
and non-contextual methods ranges from 1% to 17%. The smallest differences are usually observed when the source map has a coarser spatial resolution than the target. It is impossible to obtain high scores on a spatial super-resolution task without adding fine geometric and spatial information (e.g., high resolution images on the same extent). In practice, a good rule of thumb is to estimate that the MMU of the target maps is always of the same magnitude than the source one (i.e., translation a 25 ha MMU land-cover map results in a more or less 25 ha MMU). Conversely, significantly better results are observed when a high resolution map is translated into a coarser one.

<table><caption>Table 5.1: Agreement between translations and target maps. P: CGLS-LC100, C: CLC, O: OSO, G1: OCS-GEc, G2: OCS-GEu, M: MOS. Best values are in bold.</caption><thead><tr><th rowspan="2">Source</th><th rowspan="2">Target</th><th colspan="4">CGLS (P)</th><th colspan="4">CLC (C)</th><th colspan="4">OSO (O)</th><th colspan="4">OCS-GEc (G1)</th><th colspan="4">OCS-GEu (G2)</th><th colspan="4">MOS (M)</th><th rowspan="2">Average</th></tr><tr><th>C</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>O</th><th>G2</th><th>P</th><th>C</th><th>O</th><th>G1</th><th>P</th><th>C</th><th>O</th><th>M</th><th>A</th></tr></thead><tbody><tr><td rowspan="4">OA</td><td>SoftSem</td><td>52</td><td>42</td><td>56</td><td>70</td><td>75</td><td>65</td><td>49</td><td>67</td><td>77</td><td>79</td><td>62</td><td>59</td><td>69</td><td>76</td><td>82</td><td>56</td><td>41</td><td>34</td><td>87</td><td>57</td><td>40</td><td>31</td><td>75</td><td>80</td><td>76</td><td>59</td><td>62</td></tr><tr><td>SoftLearntGridPattern</td><td>54</td><td>44</td><td>65</td><td>70</td><td>75</td><td>68</td><td>55</td><td>71</td><td>78</td><td>79</td><td>72</td><td>62</td><td>73</td><td>80</td><td>82</td><td>63</td><td>47</td><td>49</td><td>89</td><td>62</td><td>41</td><td>41</td><td>78</td><td>84</td><td>82</td><td>62</td><td>66</td></tr><tr><td>mono-LC</td><td>65</td><td>57</td><td>69</td><td>78</td><td>76</td><td>75</td><td>59</td><td>73</td><td>80</td><td>79</td><td>77</td><td>69</td><td>80</td><td>86</td><td>85</td><td>71</td><td>58</td><td>58</td><td>93</td><td>69</td><td>54</td><td>53</td><td>79</td><td>86</td><td>84</td><td>64</td><td>72</td></tr><tr><td>multi-LC</td><td>65</td><td>57</td><td>69</td><td>78</td><td>76</td><td>74</td><td>59</td><td>73</td><td>81</td><td>79</td><td>76</td><td>68</td><td>80</td><td>86</td><td>86</td><td>71</td><td>60</td><td>58</td><td>93</td><td>70</td><td>57</td><td>54</td><td>81</td><td>86</td><td>84</td><td>64</td><td>73</td></tr><tr><td rowspan="4">mF1</td><td>SoftSem</td><td>13</td><td>17</td><td>22</td><td>15</td><td>24</td><td>46</td><td>32</td><td>36</td><td>31</td><td>42</td><td>38</td><td>19</td><td>36</td><td>20</td><td>38</td><td>27</td><td>10</td><td>17</td><td>27</td><td>20</td><td>8</td><td>8</td><td>29</td><td>38</td><td>19</td><td>19</td><td>25</td></tr><tr><td>SoftLearntGridPattern</td><td>13</td><td>18</td><td>19</td><td>16</td><td>24</td><td>47</td><td>32</td><td>33</td><td>30</td><td>42</td><td>47</td><td>24</td><td>34</td><td>20</td><td>42</td><td>37</td><td>15</td><td>20</td><td>27</td><td>28</td><td>12</td><td>10</td><td>27</td><td>43</td><td>25</td><td>18</td><td>27</td></tr><tr><td>mono-LC</td><td>30</td><td>29</td><td>30</td><td>20</td><td>33</td><td>58</td><td>37</td><td>38</td><td>30</td><td>41</td><td>61</td><td>40</td><td>45</td><td>27</td><td>53</td><td>52</td><td>34</td><td>31</td><td>43</td><td>52</td><td>29</td><td>25</td><td>40</td><td>45</td><td>30</td><td>23</td><td>38</td></tr><tr><td>multi-LC</td><td>35</td><td>30</td><td>30</td><td>20</td><td>34</td><td>57</td><td>37</td><td>37</td><td>29</td><td>41</td><td>57</td><td>41</td><td>44</td><td>27</td><td>54</td><td>53</td><td>39</td><td>34</td><td>44</td><td>50</td><td>34</td><td>27</td><td>46</td><td>48</td><td>33</td><td>24</td><td>39</td></tr><tr><td rowspan="4">EPI</td><td>SoftSem</td><td>28</td><td>5</td><td>6</td><td>5</td><td>17</td><td>30</td><td>6</td><td>6</td><td>7</td><td>20</td><td>36</td><td>36</td><td>28</td><td>32</td><td>58</td><td>28</td><td>32</td><td>29</td><td>79</td><td>32</td><td>36</td><td>29</td><td>77</td><td>54</td><td>54</td><td>38</td><td>31</td></tr><tr><td>SoftLearntGridPattern</td><td>27</td><td>5</td><td>3</td><td>5</td><td>16</td><td>30</td><td>6</td><td>5</td><td>7</td><td>20</td><td>42</td><td>35</td><td>25</td><td>31</td><td>57</td><td>35</td><td>31</td><td>30</td><td>80</td><td>35</td><td>32</td><td>30</td><td>77</td><td>57</td><td>55</td><td>38</td><td>31</td></tr><tr><td>mono-LC</td><td>22</td><td>4</td><td>3</td><td>4</td><td>12</td><td>27</td><td>6</td><td>5</td><td>6</td><td>17</td><td>39</td><td>36</td><td>25</td><td>28</td><td>49</td><td>31</td><td>29</td><td>24</td><td>77</td><td>29</td><td>30</td><td>21</td><td>70</td><td>50</td><td>55</td><td>29</td><td>28</td></tr><tr><td>multi-LC</td><td>23</td><td>4</td><td>3</td><td>4</td><td>13</td><td>28</td><td>6</td><td>4</td><td>6</td><td>18</td><td>41</td><td>38</td><td>26</td><td>51</td><td>31</td><td>31</td><td>31</td><td>25</td><td>79</td><td>30</td><td>31</td><td>23</td><td>72</td><td>52</td><td>55</td><td>29</td><td>29</td></tr></tbody></table>

mF1 differences reveal that mono and multi-land-cover map translations successfully use spatial context to outperform the simpler counterparts in terms of number of predicted classes. We provide the observed per-class f1-score in Figure 5.7. Since displaying all the 26 possible configurations would be counterproductive, we add the confusion matrices of all maps for each target, resulting in one confusion matrix per target map. We then compute the per-class f1-score, e.g. CLC per-class f1-score is computed on the fused confusion matrix of OSO-to-CLC, MOS-to-CLC, PROBA-to-CLC, OCS-GEc-to-CLC and OCS-GEu-to-CLC. In Figure 5.7, a high f1-score is reached when the translation from all sources to the considered target is successful. The well predicted classes are identical for all methods. The translation into CLC is the one for which context-wise methods are the most beneficial, as it significantly increases the number of partially predictable classes, compared to the semantic and statistical baselines. The insertion of context mainly helps specific classes, especially those defined by a spatial pattern such as CLC *Heterogeneous crops* (mix between arable and permanent crops), and on spatially correlated classes. Forests in mountainous areas mainly include coniferous stands. Thus, a forest in this area is more likely to be translated as *Coniferous* than *Broad-leaved*.

Our approach has a similar agreement to the mono-land-cover map scenario, exhibiting close scores in most cases. However, it tends to slightly under-perform on the OSO-any other configuration. This is mainly due to the fact that our MLCT-Net tends to have more difficulties in learning the MMU than the mono-land-cover map counterpart. This observation is comforted by noticing that the mean area of errors in the multi-land-cover map model is significantly smaller. This can partly be explained by the difficulty in learning the concept of MMU in a shared representation space, due to the risk of also applying
the same MMU when translating finer resolution land-cover map. One could argue that learning the MMU only requires estimating the area occupied by classes and filtering non adequate small areas. However, this would overlook that estimating areas is not a trivial task for a network fed with image patches due to the lack of information on edges (ideally, this would require processing the whole data at once, which is unfeasible). Furthermore, undetected areas in the target data act like a generalisation procedure neglecting some of the information. While this last statements affect both multi and mono-land-cover map models, the difficulty in learning the MMU naturally increases as the number of generalisation rules (and errors) increases, explaining the poorer MMU learning of the multi-land-cover map model compared to its mono-land-cover map counterpart. Since OSO is the highest resolution map used in this study, translation, from OSO are the most prone to MMU errors explaining the observed slight under-performance compared to the mono-land-cover map model.

<figure><img src="image_51.png" /><figcaption>Figure 5.7: Per-class f1 agreement computed on the sum of the translation confusion matrices of all the sources to one target.</figcaption></figure>
### 5.1.3.3 Impact of the number of input land-cover maps

We propose to analyse the influence of the number of maps fed into MLCT-Net on the quality of the translation. Figure 5.8 displays the accuracy measured on ground truth depending on the number of target maps used for learning. Each histogram represents the stacking of translation results from all three France-wide maps (CGLS, CLC, OSO) to the considered target. For instance, the first histogram presents the average translation results of CLC to CGLS and OSO to CGLS-LC100 for different models trained to perform mono-LC (1 map) or multi-LC translation using (2 to 6 maps). The three histograms OSC-GE cover, OSCGE use, and MOS are directly dependent on the network's spatial generalisation ability as they result from the average of translation performed to a broader extent than their original one. For instance, the MOS histogram is obtained by averaging France-wide translated MOS using CGLS, CLC or OSO as input, while the actual MOS only covers the Paris region. Error bars are computed as the mean of uncertainties estimated using Equation 5.2.

$$u(t) = \frac{1}{n} \sum_{s=1}^{m} z \sqrt{\frac{OA_s(1 - OA_s)}{n}}, \qquad (5.2)$$

where $u(t)$ is the uncertainty for a target map $t$, $s$ is the considered source map, $OA_s$ is the estimated accuracy of the translation from source $s$ to map $t$, $z = 1.96$ for 95% confidence, and $n$ is the ground truth sample size.

<figure><img src="image_52.png" /><figcaption>Figure 5.8: Mean accuracy per target land-cover map for different models trained with one (mono-land-cover map) up to six maps. The red-dotted line separates land-cover map available France wide (left) from those with smaller spatial extent (right).</figcaption></figure>

Our first observation is that although the model trained on six maps tends to perform better in the majority of cases, the performance variations observed on CGLS-LC100, CLC, and OSO remain insignificant given the size of the ground truth sample. This statement prevents us from concluding on a real advantage of using a multi-land-cover map model for these three maps. This observation is further supported by the fact that there is no stable trend of a performance increase when going from 2 to 6 maps. Thus additional experiments with more maps are required for further analysis. On the other hand, a more significant trend is observed on the MOS, OCS-GEc and OCS-GEu maps, which all initially covered
only a fraction of the territory. The progressive performance growth with the increasing number of maps comforts the previous analysis of greater robustness to generalisation to new landscapes of multi-LC models compared to the mono-land-cover map model.

### 5.1.3.4 Land-cover map extension

The generalisation ability of a deep neural network is a key feature when studying the representativeness of the shared space and subsequently the "universality" of such learnt representation. A universal representation should be able to represent all land cover types independently from the spatial extent enabling to use a model trained on small spatial extent maps to broader extents. This would allow to generate only high quality land-cover maps on a restricted area without spending too much time.

We propose to evaluate the ability to retrieve the target MOS, OCS-GEc and OCS-GEu over France from the sources OSO, CLC and CGLS land-cover maps, while the 3 target land-cover maps originally covers less than 20% of the country. Each source map is translated into one of the targets at a France-wide scale. The translation may face unseen classes during training in both source and target maps (e.g., there is no *glacier* on the original MOS spatial extent) resulting in wrong translations. Therefore, for each pair of source/target maps, the semantic baseline is used to translate source classes unseen during training. Unseen target classes during training are ignored. OCS-GEc *Snowfields and glaciers* and *Other non-woody formations* are considered unseen due to numerous errors in the OCS-GE data. In this setup, mono-land-cover map models cannot be trained with the geographical coordinates sub-module since they are trained solely on the original target spatial extent. To be able to assess if differences between the mono and multi-land-cover map models are due to the use of the geographical coordinates sub-module, we provide the multi-land-cover map results with and without it.

<table><caption>Table 5.2: Translation results for the 3 full France maps computed on the ground truth. "no-c" corresponds to ablation cases where the geographical coordinate sub-module is removed. X denotes impossible translations with the geographical context encoding.</caption><thead><tr><th rowspan="2">Source</th><th rowspan="2">Target</th><th colspan="5">P</th><th colspan="5">C</th><th colspan="5">O</th><th rowspan="2">Average</th></tr><tr><th>C</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>G1</th><th>G2</th><th>M</th></tr></thead><tbody><tr><td rowspan="6">OA<sub>gt</sub></td><td>SoftSem</td><td>47</td><td>46</td><td>62</td><td>79</td><td>81</td><td><strong>72</strong></td><td>51</td><td>76</td><td>84</td><td>85</td><td>71</td><td>65</td><td>86</td><td>86</td><td>92</td><td>72</td></tr><tr><td>SoftGridPattern</td><td>52</td><td>45</td><td>68</td><td>80</td><td>81</td><td>68</td><td>57</td><td>77</td><td>85</td><td>85</td><td>73</td><td>68</td><td>86</td><td>89</td><td>92</td><td>74</td></tr><tr><td>mono-LC no c</td><td>57</td><td>51</td><td>70</td><td>82</td><td>76</td><td>69</td><td>58</td><td>78</td><td><strong>86</strong></td><td>78</td><td>74</td><td>70</td><td>86</td><td>91</td><td>87</td><td>74</td></tr><tr><td>mono-LC</td><td><strong>60</strong></td><td><strong>55</strong></td><td>X</td><td>X</td><td>X</td><td>70</td><td><strong>59</strong></td><td>X</td><td>X</td><td>X</td><td>75</td><td><strong>71</strong></td><td>X</td><td>X</td><td>X</td><td>X</td></tr><tr><td>multi-LC no c</td><td>57</td><td>52</td><td>71</td><td><strong>83</strong></td><td>82</td><td>71</td><td><strong>59</strong></td><td>78</td><td><strong>86</strong></td><td><strong>86</strong></td><td><strong>78</strong></td><td>70</td><td><strong>87</strong></td><td>91</td><td><strong>93</strong></td><td>76</td></tr><tr><td>multi-LC</td><td><strong>60</strong></td><td>53</td><td><strong>74</strong></td><td><strong>83</strong></td><td><strong>83</strong></td><td>71</td><td><strong>59</strong></td><td><strong>79</strong></td><td><strong>86</strong></td><td><strong>86</strong></td><td><strong>78</strong></td><td>70</td><td><strong>87</strong></td><td><strong>92</strong></td><td><strong>93</strong></td><td><strong>77</strong></td></tr><tr><td rowspan="6">mF1<sub>gt</sub></td><td>SoftSem</td><td>12</td><td>18</td><td>29</td><td>22</td><td>26</td><td><strong>62</strong></td><td><strong>42</strong></td><td><strong>56</strong></td><td><strong>55</strong></td><td><strong>60</strong></td><td>47</td><td>22</td><td>51</td><td>30</td><td>43</td><td>38</td></tr><tr><td>SoftGridPattern</td><td>13</td><td>18</td><td>22</td><td>25</td><td>26</td><td>59</td><td>37</td><td>50</td><td>48</td><td>57</td><td>37</td><td>20</td><td>48</td><td>31</td><td>42</td><td>36</td></tr><tr><td>mono-LC no c</td><td>21</td><td>24</td><td>31</td><td>26</td><td>22</td><td>51</td><td>36</td><td>45</td><td>42</td><td>38</td><td>53</td><td>39</td><td>53</td><td>34</td><td>39</td><td>37</td></tr><tr><td>mono-LC</td><td><strong>27</strong></td><td><strong>28</strong></td><td>X</td><td>X</td><td>X</td><td>53</td><td>36</td><td>X</td><td>X</td><td>X</td><td>56</td><td><strong>45</strong></td><td>X</td><td>X</td><td>X</td><td>X</td></tr><tr><td>multi-LC no c</td><td>21</td><td>18</td><td>33</td><td><strong>27</strong></td><td>29</td><td>57</td><td>34</td><td>51</td><td>41</td><td>55</td><td>58</td><td>34</td><td><strong>57</strong></td><td>32</td><td>48</td><td>40</td></tr><tr><td>multi-LC</td><td><strong>27</strong></td><td>22</td><td><strong>37</strong></td><td><strong>27</strong></td><td><strong>30</strong></td><td>56</td><td>33</td><td>52</td><td>43</td><td>46</td><td>59</td><td>43</td><td><strong>57</strong></td><td><strong>33</strong></td><td>50</td><td><strong>41</strong></td></tr></tbody></table>

Table 5.2 presents the results computed on the ground truth. We observe that the multi-land-cover map model outperforms the mono model, especially in terms of $mF1_{gt}$. Satisfactory MOS map translation heavily relies on the coordinate sub-module (0.39
for mono-land-cover map with no coordinates, 0.48 for multi-land-cover map with no coordinates, 0.5 for multi-land-cover map with coordinates). This can be explained by the fact that the geographical context is most useful when translating unseen objects during training (such as sea). The maps with limited spatial extent, exhibiting a lower diversity of classes and objects, benefit the most from the geographical context. On the contrary, the coordinate sub-module seems less useful on the two OCS-GE maps, which perform almost the same with and without it due to training on larger and more diverse areas.

#### 5.1.3.5 Geographical context encoding

Visualising the learnt geographical encoding is crucial to better understand its effect on translation accuracy. Figure 5.9 compares the encoding using the six maps and the mono-land-cover map approach trained on the OSO-to-CLC translation. It is obtained by applying a PCA to the output of the MLP. First, the representation obtained for the multi-land-cover map approach does not seems correlate with the number of maps or the nature of maps covering each area (see Figure 3.4). Secondly, it seems that this encoding correlates well with major French geographical landscapes such as Alpes and Pyrenees mountains, the Paris basin, and the Mediterranean seashore. These results underline that the learnt geographical encoding through a multi-land-cover map approach learns to discriminate translation based on specific characteristics, by representing comparable geographic context such as high mountains in a comparable way even when they are spatially far-away. Conversely the results with the geographical encoding obtained by the mono-land-cover map model seems to be proner to learn a land-cover specific representation to compensate for local source and target errors.

<figure><img src="image_53.png" /><figcaption>Figure 5.9: PCA representation of the learnt geographical context embedding for our multi-land-cover map model (left) and the mono-land-cover map OSO to CLC model (right). One may easily delineate the main French landscapes, namely 1) Paris basin, 2) Atlantic seacoast, 3) Medium mountains, 4) High mountains, 5) Mediterranean seashore.</figcaption></figure>
### 5.1.4 Conclusion

We comprehensively investigated the potential of country-wide multi-land-cover map translation with our novel MLCT-Net model. In order to obtain higher-quality translation than models trained on specific translations or non-spatial-context-aware translation methods, we inspired ourselves by recent work on multi-task and multi-modal deep learning models. Namely, we designed a multi-encoder decoder network incorporating a three-term loss: 1) a translation loss to evaluate the quality of the land-cover map translation, 2) a self-reconstruction loss to ensure that the embedding preserves each map information, 3) a maximum distance loss on the embedding to ensure that similar features of different maps are encoded the same way to ensure high-quality results even on unseen spatial extents. Each encoder learns to project a specific map into a representation space shared between all land-cover map. Conversely, each decoder translates this shared representation space into one target land-cover map. Our key contribution is such a universal country-wide representation space which demonstrates an increase in translation generalisation.

We comprehensively evaluated our method by comparing the obtained translations to the original land-cover map and a manually annotated ground truth. Our method outperforms the standard semantic and statistical methods that only focus on exploring per-class associations instead of defining context-aware ones. The average improvement is about 9.5% in overall agreement between source and translation compared to the semantic baseline (6.2% for the statistical baseline). In contrast with the mono-land-cover map method, although we do not learn specific translation parameters, the multi-land-cover map method is only 0.4% worse in terms of overall agreement. Furthermore, statistics computed on ground truth reveal that the multi-map model outperforms the mono-map when computing the translation of maps on a spatial extent that they do not initially cover. These results demonstrate that learning a universal representation for multiple land-cover maps improves the robustness of the translation.

## 5.2 Multi-modal land-cover translation

The previous contextual translation methods considerably improve the translation compared to traditional methods. However, they fail to correctly predict many classes, as evidenced by the low average $mF1_{gt}$ (39%). We attribute this problem to the lack of information necessary for the translation in the source map. We define the notion of the *semantic gap* as the abstract measurement of the gap between the source semantic information available (the classes and context) and the information necessary to predict all target classes. For instance, translating a source map with a single agricultural class (e.g. CGLS) to a 12 agricultural classes target map (e.g. OSO) suffers from a wide semantic gap that a contextual analysis cannot fill. Similarly, we define the notion of the *geometric gap* as the abstract measurement of the gap between the local resolution of an available source
object and its target resolution. For instance, the translation of a CGLS *Forest* whose
geometric precision is about 100 meters to an OSO *Forest* whose geometric resolution
is between 10 and 20 meters suffers from a wide geometric gap. The low EPI values
particularly underline the importance of this geometric gap.

Overcoming these two gaps requires fusing additional sources of information with the
source map. A large variety of sources can theoretically be used to reduce the two gaps.
Most current literature focuses on fusing several maps with various nomenclature and
resolution. On the other hand, few studies have focused on the fusion of other data
types, such as remote-sensing images. These works generally simply use the source map
as masking data to improve a classification entirely carried out on the images, e.g. the
image pixel can only be classified as wetland if it is classified as a wetland in this reference
map [33]. They neglect the context information by relying only on nomenclature.

This section proposes to replace this "pure image classification filtered with a reference map" with a joint image and map fusion classification. Based on the experiments conducted in the previous parts, we limit the study to developing a method compatible with the MLCT-Net architecture that has shown good translation robustness even on land-cover map with small spatial extent. The nature of additional data can widely change depending on operational constraints such as the target nomenclature and resolution and the data availability. Therefore a versatile architecture is developed based on giving selectively more weight to the data the most well-suited for a given prediction.

We propose a comparison of different multi-modal sources of information to evaluate
the versatility of the proposed method. Unable to test all potential data sources, we
focus on three types of data: optical image, radar image, and Digital Elevation Models
(DEM). Those data being the most commonly used to produce land cover maps seem
well-tailored to bring semantic information. However, their capacity to fill the semantic
gap is still to be discussed as it depends directly on the amount of information that is
not already analysable using a single source map. The selected data have a resolution
close to the highest resolved map to fill the geometric gap. We deliberately exclude the
analysis of multi-temporal remote-sensing data from the study to avoid an additional layer
of complexity.

To sum up, the key contributions of this section include:

* An improved translation tool based on a versatile image and map fusion framework built on the MLCT-NET architecture.

* An evaluation of the ability to fill the semantic and resolution gap of three mainstream data sources.

The first section presents various possible image/map fusion methods based on the MLCT-
net architecture and the state-of-the-art. The second section introduces the three additional
data incorporated into our study and discusses their potential to reduce the semantic and
geometric gaps. The third section presents the experimental protocol designed to compare
the different fusion approaches and evaluate the potential of the different additional data.
Lastly, we present the results of the experiments.

5.2.1 Data fusion architecture for translation

This section presents various MLCT-Net-based architectures to fuse a source map with
remote-sensed data. First, from the literature review conducted in Section 2.4, we identify
the stage of insertion in the architecture as a critical architecture component that may
vary depending on the data and task.

Secondly, based on the observation that the relative importance of the considered additional
data and source map should vary depending on the considered translation, we propose to
investigate the attention mechanism mentioned in Section 2.4.

Lastly, inspired by the MLCT-net representation space shared between all maps, we
propose to learn a representation space shared between the additional data and the maps.
This could enable training a single network per additional data independently from the
considered translation e.g. an image is encoded through a single encoder, fused with a given
source map representation, and decoded into any land-cover map without requiring one
image encoder per translation. We term Specificity, the additional data model dependence
on the translation. A not-specific model is observed when a single model encodes the
additional data independently from the considered translation. A target-specific model
learns a different model depending on the target translation.

5.2.1.1 Fusion stage

Traditionally, three fusion strategies are identified: early/ mid/ late-stage [320]. There are
no best strategies, and the best methods depend on the data, architecture and training
task [134, 244]. As Land-cover translation has rarely been studied no previous work can
give insight on the strategy to adopt: we propose to compare those three strategies.

The **Early** fusion approach (see Figure 5.10) is constant across all literature and always
consists of a simple concatenation of the LC with the image before feeding it to the encoder.
This approach assumes that features from the LC and the images are similar. In our case,
we expect this method to perform poorly as CNN on images intensely focuses on texture,
while low-texture land-cover maps are rich in terms of geometric information.

The **Mid** fusion (see Figure 5.12) consist in processing the map and additional data
in separate encoders to fuse progressively shared features. Plethoric solutions have
been proposed. Those presented in this section are based on solutions known to perform
reasonably for multiple tasks. The **Mid-1** is based on the Fuse-net architecture [113], easily
adaptable to work with the U-NET encoder architecture by simply adding a max-pooling
layer to the image encoder. It processes the image and map representations in parallel in
two identical architectures, fusing the image representation with a simple addition at each
encoding block of the U-Net. The **Mid-2** architecture improves the Fuse-net architecture by replacing the addition with a simple channel-wise attention module based on the Squeeze-and-Excite (SE) block (described separately in Section 5.2.1.2). Coincidentally, this makes the methods comparable to the Multi-Modal Transfer Module architecture (MMTM) proposed by [156]. Lastly, we introduce **Mid-3**, which fuses higher-level features in the decoder part of the U-Net using the same SE block as **Mid-2**.

**Late** fusion methods (see Figure 5.11) consist of training simultaneously one network per modality and either concatenating or summing their results just before the last classification layer. We use the same architecture for learning the representations, as commonly performed when the two modalities are identical. Like **Mid-2** and **Mid-3** architectures, we propose to fuse the different representations using SE.

<figure><img src="image_54.png" /><figcaption>Figure 5.10: Early fusion strategy (purple box). For convenience, only one branch of the MLCT-Net is presented.</figcaption></figure>

<figure><img src="image_55.png" /><figcaption>Figure 5.11: Late fusion strategy (purple box). For convenience, only one branch of the MLCT-Net is presented.</figcaption></figure>
<figure><img src="image_56.png" /><figcaption>Figure 5.12: Three different intermediate fusion strategies. For convenience, only one branch of the MLCT-Net is presented.</figcaption></figure>

<figure><img src="image_57.png" /><figcaption>Figure 5.13: Squeeze and excite block [131].</figcaption></figure>
### 5.2.1.2 Channel attention

A current trend in computer vision is to incorporate attention mechanisms into the networks, i.e. methods to give more importance to some features than others. We distinguish two sorts of mechanisms based on the nature of the features considered: (i) spatial attention, when features are considered spatially (ii) channel-wise attention, when features are considered per-pixel. Spatial attention, for instance, implemented in Vision transformers [70], leverages the spatial sequence of pixels to give more importance to some parts of the feature maps. In our case, we deliberately ignore those methods as they require vast amounts of training data to achieve satisfying results. Moreover, as they are memory-consuming, training simultaneously with the six land-cover maps is unfeasible. On the other hand, Channel-wise attention appears very interesting, given the properties of the MLCT-Net. Indeed, different channels of the MLCT-net common representation space encode for different land-cover types (see Figure 5.4). When fusing image and map features at the common embedding stage (Late fusion), it could be beneficial to give more influence to some image or map representation channels depending on their reliability on the land cover encoded by the considered channel. The experimented channel attention mechanism relies on Squeeze-and-Excite introduced in [131]. It consists of a three-step procedure presented in Figure 5.13.: i) a global average pooling computes separately the per-channel mean for the image and map representation ii) the two per-channel means are concatenated and processed by a linear layer outputting separate per channel weights for the image and map representation iii) the image and map representation channels are multiplied separately by the previous weights and then added together.

### 5.2.1.3 Image representation specificity

MLCT-Net aims to perform multiple translations using a single model. Its core idea is to translate one source map into multiple target ones and thus can be seen as a multi-task network. A core question when fusing multi-modal data into a multi-task model relies on the dependency between each task and the multi-modal data. This requires answering the question "should the multi-modal fusion model be dependent on the task ?". For instance, when fusing an image with a source land-cover map, should the model used to encode the image be the same if the source land-cover is A or B? Should it be the same if the target land cover is A or B ? To our knowledge, no previous works have been conducted to evaluate this multi-modal multi-task fusion in the remote sensing field. However, some works in the medical imagery field suggest that training a single multi-modal model to perform multiple tasks can outperform training a separate multi-modal model per task [357].

The learnt representation can be distinguished into four levels of specificity. We illustrate this idea with the following example: we target translating LC A, B and C into one another and use an optical image as additional data.

Level-1 learns a unique image representation, i.e. a unique image representations is
combined with the representations of A, B, and C. We refer to this method as the **Unique** method as a unique universal image representation is learnt.

Level-2 learns one image representation per source land-cover map, i.e. the image representation combined with the source map A representation is different from the one combined with the B or C representation. We refer to this method as the **SourceSpe** method, as one specific image model is learnt for each source map.

Level-3 learns one image representation per target land-cover map, i.e. the image representation combined with the A representation is not the same if the goal is to obtain B or C. The image representation fused with A and with B is identical if the target for A and B is C. We refer to this method as the **TargetSpe** method, as one specific image model is learnt for each target map.

Level-4 learns one image representation per source/target pairs. We do not present level four results implying training $n \times (n-1)$ image models.

## 5.2.2 Remote-sensing data sources

Our experiments incorporate three different sources of remote-sensing data, presented in Section 3.4, to alleviate the semantic and geometric gaps. Data are standardised using their respective mean and standard deviation.

### 5.2.2.1 Optical imagery: Sentinel-2

The first source is optical images. As land-cover maps are mainly produced by analysing optical images, they should be the best fitted to increase land-cover translation ability. They can both bring semantic and geometric information. For instance, such images could help retrieve the previously mentioned OSO agricultural classes. However, we underline that this data is insufficient to achieve perfect translation, as (i) we only process mono-temporal images, (ii) multiple land-cover maps such as CLC or OCSGE cover and use are not solely based on image analysis but also stem from fusing with independent databases. The core question is to identify to which extent a single image provides complementary information to source map. All experiments are conducted on Sentinel-2 cloudless synthesis.

### 5.2.2.2 Radar imagery: Sentinel-1

The second source is Synthetic-Aperture Radar imagery which offers some interesting properties. In particular, the high wavelength used in radar acquisition conveys a different radiometric information that could partially fill the semantic gap in classes related to water, urban areas, or glaciers. From an operational point of view, SAR images are more readily available for a task involving a mapping with temporal constraints or under tropical
conditions as they are insensible to weather conditions. All experiments are conducted on
the Sentinel-1 mosaic averaged temporally to reduce speckle noise.

5.2.2.3 Digital Elevation Model: ALOS World 3D

The last source is a Digital Elevation Model (DEM). DEM provides interesting semantic
information for very specific classes correlated to topographic specificities (e.g. conifers and
snow are often observed at high altitudes). It could also help fill the geometric information
for classes exhibiting a geometry correlated to topography. For instance, tree density
might change abruptly on the ridge of mountains (due to both luminosity and humidity
variations on both sides), resulting in a change in land-cover classification. All experiments
are conducted on ALOS World 3D and the related features computed (exposure, TPI,
roughness).

5.2.3 Experimental protocol

5.2.3.1 Finding the best architecture

Many architectures can be obtained by combining the Fusion stage and Specificity char-
acteristics presented earlier. We test all possible Fusion stage - Specificity combinations
to determine the best configuration. For instance, we test the **Late** fusion either with
a **Unique**, **SourceSpe** or **TargetSpe** specificity. We underline that the **Early** fusion
methods are always Source Specific as the image and source map are concatenated from the
start and thus can not be analysed separately. This results in 13 network configurations: 1
for **Early** and 3 for **Mid-1**, **Mid-2**, **Mid-3**, **Late**.

As we aim to obtain a versatile architecture usable with other data than the three
experimented, we consider best architecture the one that obtains the best results on
average across the three additional data. Subsequently, we test each architecture with
each additional data separately, *i.e.* the 13 configurations are trained using optical images,
SAR and DEM, resulting in 39 models.

5.2.3.2 Assessing the potential of multi-modal translation

Multi-modal translations should obtain a higher quality than using only the map or
classifying the remote-sensed data without any source map information. We compute
the classification of the additional data without any source map using the same U-Net
architecture as the one used for fusion: we train to classify the S1 and S2 data into each
of the target maps (the DEM-only classification is not presented as it gives poor results).
We underline that a separate model is trained for each of the specific spatial extents of the
dataset. For instance, three models are trained to classify S2 into CLC, one France-wide
(corresponding to OSO to CLC or CGLS to CLC translation), one on the OCSGE spatial
extent, and one on the MOS spatial extent.

## 5.2.4 Results

### 5.2.4.1 Finding the best architecture

Table 5.3 present a comparative study of the 13 configurations in terms of OA<sub>gt</sub> using Sentinel-2 data. The fact that the same conclusion can be drawn using the other additional data (see Appendix I). A first observation is that the specificity does not affect the results for a given fusion stage. This underlines that training a unique model, independent from the considered translation, is not detrimental compared to training multiple independent ones. This is interesting for operational cases, as training a single model is more time efficient. It indicates that a single model can learn to map remote-sensed images into a universal representation space.

Moreover, we observe that the Late fusion always performs better than the other configurations. This statement holds for the three data. We hypothesise that this could be attributed to the necessity to considerably transform the map and image features to make them complementary, preventing fusion at earlier stages. In the following section we only consider the Late - Unique method as it obtains the best results in the least computationally expensive way.

<table><caption>Table 5.3: Comparison of various Fusion strategies using a Sentinel-2 cloudless synthesis, the 6 land cover maps and the MLCT-net in terms of OA<sub>ag</sub>.</caption><thead><tr><th rowspan="2">Source</th><th rowspan="2">Target</th><th colspan="4">CGLS (P)</th><th colspan="4">CLC (C)</th><th colspan="4">OSO (O)</th><th colspan="4">OCSGEc (G1)</th><th colspan="4">OCSGEu (G2)</th><th colspan="4">MOS (M)</th><th rowspan="2">Average</th></tr><tr><th>C</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>O</th><th>G2</th><th>P</th><th>C</th><th>O</th><th>G1</th><th>P</th><th>C</th><th>O</th><th>MOS</th><th>C</th><th>O</th></tr></thead><tbody><tr><td rowspan="2">Early</td><td>SourceSpe</td><td>68</td><td>70</td><td>78</td><td>85</td><td>85</td><td>77</td><td>70</td><td>79</td><td>86</td><td>85</td><td>76</td><td>67</td><td>79</td><td>86</td><td>86</td><td>72</td><td>61</td><td>66</td><td>92</td><td>72</td><td>60</td><td>66</td><td>84</td><td>87</td><td>85</td><td>70</td><td>77</td></tr><tr><td>Unique</td><td>68</td><td>70</td><td>78</td><td>85</td><td>85</td><td>77</td><td>70</td><td>79</td><td>86</td><td>85</td><td>76</td><td>67</td><td>80</td><td>87</td><td>86</td><td>73</td><td>60</td><td>66</td><td>92</td><td>72</td><td>60</td><td>66</td><td>84</td><td><strong>87</strong></td><td><strong>85</strong></td><td>71</td><td>77</td></tr><tr><td rowspan="3">Mid-1</td><td>SourceSpe</td><td>67</td><td>69</td><td>78</td><td>85</td><td>85</td><td>76</td><td>69</td><td>78</td><td>85</td><td>85</td><td>75</td><td>66</td><td>80</td><td>87</td><td>86</td><td>72</td><td>59</td><td>66</td><td>92</td><td>72</td><td>59</td><td>65</td><td>84</td><td><strong>87</strong></td><td>84</td><td>71</td><td>76</td></tr><tr><td>TargetSpe</td><td>67</td><td>69</td><td>78</td><td>86</td><td>85</td><td>77</td><td>70</td><td>79</td><td>86</td><td>85</td><td>76</td><td>67</td><td>80</td><td>86</td><td>86</td><td>72</td><td>60</td><td>67</td><td>92</td><td>73</td><td>60</td><td>66</td><td>84</td><td><strong>87</strong></td><td><strong>85</strong></td><td>71</td><td>77</td></tr><tr><td>Unique</td><td>67</td><td>69</td><td>77</td><td>85</td><td>85</td><td>77</td><td>70</td><td>79</td><td>86</td><td>85</td><td>76</td><td>67</td><td>80</td><td>86</td><td>86</td><td>73</td><td>60</td><td>67</td><td>92</td><td>72</td><td>58</td><td>65</td><td>82</td><td><strong>87</strong></td><td><strong>85</strong></td><td>71</td><td>76</td></tr><tr><td rowspan="3">Mid-2</td><td>SourceSpe</td><td>67</td><td>70</td><td>78</td><td>85</td><td>85</td><td>77</td><td>69</td><td>79</td><td>86</td><td>84</td><td>77</td><td>67</td><td>80</td><td>86</td><td>86</td><td>72</td><td>60</td><td>68</td><td>92</td><td>72</td><td>59</td><td>65</td><td>84</td><td>86</td><td><strong>85</strong></td><td>71</td><td>77</td></tr><tr><td>TargetSpe</td><td>67</td><td>69</td><td>78</td><td>85</td><td>85</td><td>77</td><td>69</td><td>80</td><td>86</td><td>84</td><td>76</td><td>67</td><td>80</td><td>87</td><td>86</td><td>73</td><td>60</td><td>68</td><td>92</td><td>72</td><td>58</td><td>65</td><td>83</td><td>86</td><td><strong>85</strong></td><td>71</td><td>76</td></tr><tr><td>Unique</td><td>68</td><td>71</td><td>79</td><td>86</td><td>86</td><td>78</td><td>70</td><td>79</td><td>86</td><td>86</td><td>77</td><td>68</td><td>81</td><td>87</td><td>86</td><td>74</td><td>61</td><td>68</td><td>93</td><td>73</td><td>59</td><td>66</td><td>85</td><td>86</td><td><strong>85</strong></td><td>71</td><td>77</td></tr><tr><td rowspan="3">Mid-3</td><td>SourceSpe</td><td>67</td><td>71</td><td>79</td><td>86</td><td>86</td><td>77</td><td>70</td><td>80</td><td>86</td><td>85</td><td>77</td><td>67</td><td>81</td><td>86</td><td>86</td><td>72</td><td>61</td><td>69</td><td>92</td><td>73</td><td>59</td><td>66</td><td>85</td><td>86</td><td><strong>85</strong></td><td>71</td><td>77</td></tr><tr><td>TargetSpe</td><td>68</td><td>71</td><td>79</td><td>86</td><td>86</td><td>77</td><td>70</td><td>79</td><td>86</td><td>84</td><td>77</td><td>67</td><td>80</td><td>86</td><td>86</td><td>73</td><td>61</td><td>68</td><td>92</td><td>73</td><td>59</td><td>66</td><td>84</td><td><strong>87</strong></td><td><strong>85</strong></td><td>71</td><td>77</td></tr><tr><td>Unique</td><td><strong>71</strong></td><td><strong>73</strong></td><td><strong>81</strong></td><td><strong>87</strong></td><td><strong>86</strong></td><td><strong>79</strong></td><td><strong>73</strong></td><td><strong>81</strong></td><td><strong>87</strong></td><td><strong>86</strong></td><td><strong>78</strong></td><td><strong>70</strong></td><td><strong>82</strong></td><td><strong>88</strong></td><td><strong>87</strong></td><td><strong>75</strong></td><td><strong>64</strong></td><td><strong>70</strong></td><td><strong>93</strong></td><td><strong>75</strong></td><td><strong>64</strong></td><td><strong>70</strong></td><td><strong>87</strong></td><td><strong>87</strong></td><td><strong>85</strong></td><td><strong>72</strong></td><td><strong>79</strong></td></tr><tr><td rowspan="2">Late</td><td>SourceSpe</td><td><strong>71</strong></td><td><strong>73</strong></td><td><strong>81</strong></td><td><strong>87</strong></td><td><strong>86</strong></td><td><strong>79</strong></td><td><strong>73</strong></td><td><strong>81</strong></td><td><strong>87</strong></td><td><strong>86</strong></td><td><strong>77</strong></td><td><strong>69</strong></td><td><strong>82</strong></td><td><strong>88</strong></td><td><strong>87</strong></td><td><strong>75</strong></td><td><strong>63</strong></td><td><strong>70</strong></td><td><strong>93</strong></td><td><strong>75</strong></td><td><strong>63</strong></td><td><strong>69</strong></td><td><strong>86</strong></td><td><strong>87</strong></td><td><strong>85</strong></td><td><strong>72</strong></td><td><strong>79</strong></td></tr><tr><td>TargetSpe</td><td><strong>71</strong></td><td><strong>73</strong></td><td><strong>81</strong></td><td><strong>87</strong></td><td><strong>86</strong></td><td><strong>78</strong></td><td><strong>73</strong></td><td><strong>81</strong></td><td><strong>87</strong></td><td><strong>86</strong></td><td><strong>77</strong></td><td><strong>70</strong></td><td><strong>82</strong></td><td><strong>88</strong></td><td><strong>87</strong></td><td><strong>75</strong></td><td><strong>64</strong></td><td><strong>70</strong></td><td><strong>93</strong></td><td><strong>74</strong></td><td><strong>63</strong></td><td><strong>69</strong></td><td><strong>86</strong></td><td><strong>87</strong></td><td><strong>85</strong></td><td><strong>72</strong></td><td><strong>79</strong></td></tr></tbody></table>

### 5.2.4.2 Comparing the remote-sensed data sources

Figure 5.14 compares the results obtained for the three different remote-sensed images using the Late - Unique. We first observe that for the coarse-to-fine resolution case, additional data sources considerably increases the qualitative aspect of the result compared to the pure map translation approach. This is, for example, well illustrated by the first and last row of Figure 5.14 (respectively, CLC to MOS and CGLS to OCSGEu translation). When the source resolution is equivalent to or finer than the target one, more data only marginally increases the semantic accuracy, e.g. the S2 mosaics improves the MOS to OSO translation by helping to retrieve agricultural classes. We observed that
fusion gives significantly different results than using each separately. For instance, in the CGLS to OCSGEu translation, the fusion-based approach is the only one able to retrieve 411: "Road Networks".

Those results are comforted by the analysis of the OA<sub>ag</sub> and mF1<sub>ag</sub> (Table 5.4). In particular, the observed average +2% OA<sub>ag</sub> and mF1<sub>ag</sub> increase between the map+S2 and S2 only strategy encompass widely heterogeneous per-translation results. When the source map is CGLS or CLC (the coarsest maps), the difference between using only S2 or fusing S2 with one of those two maps is almost nonexistent and slightly detrimental (-0.6% mF1<sub>ag</sub>) due to the difficulty of optimising the MLCT-Net: coarse information is often of low interest when predicting a fine-resolved map, as one source object might encompass multiple target land-cover types. Conversely, when the source map is identically or more resolved than the target, the fusion performs significantly better than using the image-alone strategy (+4.6% mF1<sub>ag</sub>). We conclude that land-cover to additional data fusion is only relevant when the source resolution is comparable or higher than the target.

Another observation is that fusing S1 still performs reasonably well and is often preferable to the S2-only approach. Therefore it appears possible to perform our land-cover fusion strategy even in areas where optical imagery suffers from cloud occlusion.

The OA<sub>ag</sub> and mF1<sub>ag</sub> of the map-only translation approach are always lower or equivalent to the S2-only strategy. Consequently, land-cover translation could appear inefficient for operational use as using a single image often outperforms the land-cover translation approach. However, we argue that this higher OA<sub>ag</sub> and mF1<sub>ag</sub> is mostly due to a higher capability of the network to replicate target errors when using an image as source data instead of the map. This behaviour is demonstrated in Table 5.5 using the ground truth. We observe that the map-only approach often exhibits higher metrics than the S2 only, especially when the fine-resolved OSO map is translated into one of the others. Furthermore this two overall metrics encompass significant per-class differences, illustrated in Figure 5.15. The map-only translation approach significantly outperforms the S2-only approach for classes with various semantic definitions such as CGLS 70: "Snow and Ice", CLC 212: "Permanently irrigated land", OCSGEc 1112: "Undeveloped areas". Secondly, we observe that fusion strategies seem to keep "the best of the two worlds" when either the map or the additional data obtain results used alone while the other performs well as well illustrated by the OCSGEc 1112 class example.

Additionally, Table 5.5 shows that the fusion between the image and map is most of the time beneficial compared to using each data separately exception made of the translation into the MOS map. As mentioned previously, those measurements on the France-wide ground truth test in the case of the translation into MOS, the ability of the method to spatially generalise from an original MOS extent. We conclude that the proposed fusing method does not guarantee efficient spatial generalisation ability, i.e. the universal image+map representation space is non-homogenous across the territory preventing spatial generalisation.
Lastly, we observe that the S1 and DEM data are rarely more beneficial than the S2 data except on very specific classes such as "Snow and Ice" equivalent classes for CGLS, CLC and OSO.

<figure><img src="image_58.png" /><figcaption>Figure 5.14: Comparison of the translation results using various remote-sensed images and the Late-Unique fusion strategy. Map, s1 and s2 results are obtained using respectively only the source map, only the Sentinel-1 data, and only the Sentinel-2 data. map+s2 correspond to results obtained by fusing the image and map representation using the Late-Unique fusion strategy.</figcaption></figure>
<table><caption>Table 5.4: OA<sub>ag</sub> and mF1<sub>ag</sub> comparison for various additional data fusing. Grey columns denotes translation for which the source map is higher resolved than the target one</caption><thead><tr><th rowspan="2">Source</th><th rowspan="2">Target</th><th colspan="4">CGLS (P)</th><th colspan="4">CLC (C)</th><th colspan="4">OSO (O)</th><th colspan="4">OCSGEc (G1)</th><th colspan="4">OCSGEu (G2)</th><th colspan="4">MOS (M)</th><th rowspan="2">Total average</th></tr><tr><th>C</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>O</th><th>G1</th><th>G2</th><th>P</th><th>C</th><th>O</th><th>G1</th><th>G2</th><th>P</th><th>C</th><th>O</th><th>G1</th></tr></thead><tbody><tr><td rowspan="7">OA</td><td>Map only</td><td>65</td><td>57</td><td>69</td><td>78</td><td>76</td><td>75</td><td>59</td><td>73</td><td>80</td><td>79</td><td>77</td><td>69</td><td>80</td><td>86</td><td>85</td><td>71</td><td>58</td><td>58</td><td>93</td><td>69</td><td>54</td><td>53</td><td>79</td><td>86</td><td>84</td><td>64</td><td>72</td></tr><tr><td>S1 only</td><td>69</td><td>67</td><td>77</td><td>84</td><td>83</td><td>77</td><td>67</td><td>77</td><td>84</td><td>83</td><td>77</td><td>69</td><td>77</td><td>84</td><td>83</td><td>71</td><td>57</td><td>59</td><td>85</td><td>71</td><td>57</td><td>59</td><td>77</td><td>86</td><td>82</td><td>65</td><td>74</td></tr><tr><td>S2 only</td><td>70</td><td><strong>74</strong></td><td><strong>81</strong></td><td><strong>87</strong></td><td><strong>86</strong></td><td>77</td><td><strong>74</strong></td><td><strong>81</strong></td><td><strong>87</strong></td><td><strong>86</strong></td><td>77</td><td><strong>70</strong></td><td>81</td><td>87</td><td>86</td><td>74</td><td>60</td><td>68</td><td>87</td><td>74</td><td>60</td><td>68</td><td>80</td><td>86</td><td>83</td><td>70</td><td>77</td></tr><tr><td>Map + DEM</td><td>64</td><td>59</td><td>72</td><td>80</td><td>80</td><td>73</td><td>60</td><td>74</td><td>82</td><td>81</td><td>76</td><td>69</td><td>80</td><td>87</td><td>86</td><td>72</td><td>61</td><td>60</td><td>93</td><td>71</td><td>58</td><td>56</td><td>82</td><td>86</td><td>84</td><td>64</td><td>73</td></tr><tr><td>Map + Aspect</td><td>66</td><td>57</td><td>70</td><td>79</td><td>77</td><td>76</td><td>60</td><td>73</td><td>81</td><td>80</td><td>78</td><td>68</td><td>80</td><td>86</td><td>86</td><td>73</td><td>61</td><td>59</td><td>93</td><td>72</td><td>57</td><td>55</td><td>81</td><td><strong>87</strong></td><td>84</td><td>64</td><td>73</td></tr><tr><td>Map + DEM + Aspect</td><td>67</td><td>60</td><td>72</td><td>80</td><td>81</td><td>75</td><td>60</td><td>74</td><td>81</td><td>82</td><td>79</td><td>69</td><td>80</td><td>86</td><td>86</td><td>74</td><td>61</td><td>60</td><td>93</td><td>73</td><td>59</td><td>55</td><td>81</td><td><strong>87</strong></td><td>84</td><td>63</td><td>74</td></tr><tr><td>Map + S1</td><td>70</td><td>68</td><td>79</td><td>86</td><td>85</td><td>78</td><td>68</td><td>79</td><td>86</td><td>85</td><td>77</td><td><strong>70</strong></td><td><strong>82</strong></td><td><strong>88</strong></td><td><strong>87</strong></td><td>73</td><td>63</td><td>65</td><td>93</td><td>73</td><td>62</td><td>63</td><td>85</td><td><strong>87</strong></td><td><strong>85</strong></td><td>69</td><td>77</td></tr><tr><td rowspan="7">mF1</td><td>Map only</td><td>30</td><td>29</td><td>30</td><td>20</td><td>33</td><td>58</td><td>37</td><td>38</td><td>30</td><td>41</td><td>61</td><td>40</td><td>45</td><td>27</td><td>53</td><td>52</td><td>34</td><td>31</td><td>43</td><td>52</td><td>29</td><td>25</td><td>40</td><td>45</td><td>30</td><td>23</td><td>38</td></tr><tr><td>S1 only</td><td>39</td><td>44</td><td>39</td><td>26</td><td>43</td><td>55</td><td>44</td><td>39</td><td>26</td><td>43</td><td>55</td><td>38</td><td>39</td><td>26</td><td>43</td><td>50</td><td>28</td><td>32</td><td>27</td><td>50</td><td>28</td><td>32</td><td>38</td><td>46</td><td>27</td><td>29</td><td>36</td></tr><tr><td>S2 only</td><td>43</td><td><strong>53</strong></td><td><strong>48</strong></td><td><strong>36</strong></td><td><strong>58</strong></td><td>59</td><td><strong>53</strong></td><td><strong>48</strong></td><td><strong>36</strong></td><td><strong>58</strong></td><td>59</td><td>43</td><td>48</td><td>36</td><td>58</td><td>54</td><td>36</td><td><strong>45</strong></td><td>37</td><td>54</td><td>36</td><td><strong>45</strong></td><td>48</td><td>50</td><td>40</td><td>35</td><td>47</td></tr><tr><td>Map + DEM</td><td>35</td><td>35</td><td>34</td><td>20</td><td>40</td><td>58</td><td>37</td><td>39</td><td>29</td><td>45</td><td>58</td><td>41</td><td>47</td><td>28</td><td>55</td><td>54</td><td>38</td><td>35</td><td>44</td><td>52</td><td>37</td><td>30</td><td>49</td><td>50</td><td>39</td><td>24</td><td>41</td></tr><tr><td>Map + Aspect</td><td>35</td><td>30</td><td>32</td><td>20</td><td>35</td><td>61</td><td>37</td><td>37</td><td>24</td><td>43</td><td>63</td><td>40</td><td>44</td><td>23</td><td>53</td><td>55</td><td>39</td><td>33</td><td>35</td><td>53</td><td>35</td><td>27</td><td>47</td><td>49</td><td>37</td><td>23</td><td>39</td></tr><tr><td>Map + DEM + Aspect</td><td>35</td><td>35</td><td>33</td><td>21</td><td>41</td><td>62</td><td>38</td><td>40</td><td>28</td><td>46</td><td>62</td><td>42</td><td>48</td><td>28</td><td>54</td><td>56</td><td>38</td><td>36</td><td>44</td><td>54</td><td>37</td><td>29</td><td>48</td><td>50</td><td>40</td><td>25</td><td>41</td></tr><tr><td>Map + S1</td><td>41</td><td>43</td><td>42</td><td>26</td><td>52</td><td>60</td><td>46</td><td>43</td><td>31</td><td>54</td><td>60</td><td>43</td><td>47</td><td>27</td><td>57</td><td>55</td><td>41</td><td>39</td><td>41</td><td>54</td><td>40</td><td>35</td><td>53</td><td><strong>51</strong></td><td><strong>44</strong></td><td>32</td><td>45</td></tr><tr><td rowspan="7">mF1<sub>gt</sub></td><td>Map + S2</td><td><strong>44</strong></td><td>50</td><td>47</td><td>30</td><td>57</td><td><strong>65</strong></td><td>52</td><td><strong>48</strong></td><td><strong>36</strong></td><td>57</td><td><strong>64</strong></td><td><strong>47</strong></td><td><strong>50</strong></td><td><strong>38</strong></td><td><strong>60</strong></td><td><strong>62</strong></td><td><strong>44</strong></td><td><strong>45</strong></td><td><strong>49</strong></td><td><strong>61</strong></td><td><strong>44</strong></td><td>44</td><td><strong>57</strong></td><td>49</td><td><strong>44</strong></td><td><strong>36</strong></td><td><strong>49</strong></td></tr></tbody></table>

<table><caption>Table 5.5: OA<sub>gt</sub> and mF1<sub>gt</sub> comparison for the map only, s2 only, map+s2.</caption><thead><tr><th rowspan="2">Source</th><th rowspan="2">Target</th><th colspan="4">CGLS (P)</th><th colspan="4">CLC (C)</th><th colspan="4">OSO (O)</th><th colspan="4">OCSGEc (G1)</th><th colspan="4">OCSGEu (G2)</th><th colspan="4">MOS (M)</th><th rowspan="2">Average</th></tr><tr><th>C</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>O</th><th>G1</th><th>G2</th><th>P</th><th>C</th><th>O</th><th>G1</th><th>G2</th><th>P</th><th>C</th><th>O</th><th>G1</th></tr></thead><tbody><tr><td rowspan="3">OA<sub>gt</sub></td><td>Map</td><td>60</td><td>53</td><td>74</td><td>83</td><td>83</td><td><strong>71</strong></td><td>59</td><td>79</td><td>86</td><td><strong>86</strong></td><td><strong>78</strong></td><td>70</td><td>87</td><td><strong>92</strong></td><td><strong>93</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td></tr><tr><td>s2</td><td>63</td><td>70</td><td>82</td><td>89</td><td>81</td><td>69</td><td>70</td><td>82</td><td>89</td><td>81</td><td>69</td><td>63</td><td>82</td><td>89</td><td>81</td><td>69</td><td>63</td><td>82</td><td>89</td><td>81</td><td>69</td><td>63</td><td>82</td><td>89</td><td>81</td><td>69</td><td>63</td><td>82</td><td>89</td><td>81</td></tr><tr><td>Map+s2</td><td><strong>70</strong></td><td><strong>72</strong></td><td><strong>84</strong></td><td><strong>91</strong></td><td>67</td><td><strong>71</strong></td><td><strong>72</strong></td><td><strong>85</strong></td><td><strong>91</strong></td><td>66</td><td>72</td><td><strong>72</strong></td><td><strong>88</strong></td><td><strong>92</strong></td><td>69</td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td><td><strong>77</strong></td></tr><tr><td rowspan="3">mF1<sub>gt</sub></td><td>Map</td><td>27</td><td>22</td><td>37</td><td>27</td><td><strong>30</strong></td><td>56</td><td>33</td><td><strong>52</strong></td><td><strong>43</strong></td><td><strong>46</strong></td><td><strong>59</strong></td><td>43</td><td><strong>57</strong></td><td>33</td><td><strong>50</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td><td><strong>41</strong></td></tr><tr><td>S2 only</td><td>37</td><td>40</td><td>46</td><td>31</td><td>39</td><td>52</td><td>40</td><td>46</td><td>31</td><td>39</td><td>52</td><td>37</td><td>46</td><td>31</td><td>39</td><td>52</td><td>37</td><td>46</td><td>31</td><td>39</td><td>52</td><td>37</td><td>46</td><td>31</td><td>39</td><td>52</td><td>37</td><td>46</td><td>31</td></tr><tr><td>Map+s2</td><td><strong>44</strong></td><td><strong>41</strong></td><td><strong>47</strong></td><td><strong>37</strong></td><td><strong>30</strong></td><td><strong>58</strong></td><td><strong>48</strong></td><td>50</td><td>42</td><td>29</td><td><strong>59</strong></td><td><strong>47</strong></td><td>55</td><td><strong>43</strong></td><td>30</td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td><td><strong>44</strong></td></tr></tbody></table>

### 5.2.5 Conclusion

We presented a strategy to fuse remote-sensed images and map representation to both improve results and study the potential of re-using existing land-cover maps as input of the traditional image-to-land-cover classification procedure. As we consider that both the image and map should be fused using a context-aware method to increase the amount of information available, we proposed to adapt the MLCT-Net. The first section introduced how to fuse additional data inside our MLCT-net architecture. From our experiment we observe that the fusion should be performed at the land-cover representation space level. Since the land-cover representation discriminates different land-cover using different dimensions, the land-cover and image representation dimensions should be weighted differently depending on their content. Moreover, learning a single representation space for the additional data, independent from the source and target map, appears sufficient. Secondly, we focused on the impact of different sources on the quality of land cover translation. Experiments demonstrate that fusing additional data mainly the quality of translation for all source/target map couples. However, we also pointed out that the difference between classifying the remote-sensed data alone and fusing it not significantly improve the results when the source is significantly coarser than the target. Therefore we argue that fusion between image and representation should mainly be conducted using a source map with an equivalent or finer resolution than the target. Note that those
<figure><img src="image_59.png" /><figcaption>Figure 5.15: Highest observed per class $F1_{ag}$ for each source to target translation.</figcaption></figure>
conclusions are drawn from experiments on a limited number of maps and that other
factors could have played a role. Especially one could argue that the fact that results using
the remotely sensed image and low-resolved map fusion are not better than just using the
image could be a mere coincidence. Instead, one could argue that the core reason lies in
the complementary between the nomenclature of the source and target maps, e.g. CLC
to OSO translation does not give bad results because CLC has a coarse resolution but
because CLC nomenclature brings no more information to obtain OSO than the image.
However, we found it unlikely as the ≈ 50% mF1 obtained by classifying the S2 image into
CLC or CGLS indicates that many classes are not obtainable from the image data alone.
Lastly, we observed that the S2 imagery always achieved the best results amongst the
three additional data experimented on, both used alone or merged. However, in multiple
cases, the difference with S1 images remains modest, which underlines the possibility of
performing this on areas where S2 images suffer from significant cloud occlusion.

5.3 Conclusion

This chapter focused on improving the translation methods to make them usable in
multiple operational cases. Two research axis are developed: i) building a multi-translation
framework to enable predicting multiple land-cover maps translation from a single source
while ensuring good spatial generalisation properties and ii) fusing additional data to
increase the quality of the translation.

The A-UNet trained to perform a single translation suffered both from a lack of spatial generalisation and was sensible to the noise in the source and, more importantly, in the target map. To alleviate those issues, we proposed to comprehensively investigate multi-land-cover map translation with our novel MLCT-Net model driven by the observation that models trained to perform multi-lingual translation outperformed models trained to perform each language separately. In order to increase the translation ability of our model on maps with small spatial extent by reducing both over-fitting and increasing the spatial generalizability, a multi-encoder decoder network was designed, incorporating a three-term loss. A universal country-wide representation space of the six land-cover map was obtained and demonstrated good translation results compared to training separate A-UNet. In particular, the multi-land-cover map model obtained a higher OA<sub>gt</sub> and mF1<sub>gt</sub> (+2 and +3% respectively) than the mono-land cover baselines while achieving almost the same scores in terms of OA<sub>ag</sub> and mF1<sub>ag</sub> demonstrate better noise robustness. Moreover, the multi-LC training enables significantly increased cartographic generalisation ability (on average +7% OA<sub>gt</sub> and +9% mF1<sub>gt</sub> when translating CLC, CGLS or OSO into MOS).

Secondly, we investigated the potential of fusing additional data sources to increase the
quality of the translation, especially in the case where the source map is coarser than the
target. Land-cover maps are rarely used as raw data to predict other land-cover maps and
are not jointly analysed with images. We proposed to adapt our multi-land-cover map
model to perform this fusion. In particular, learning a single universal image representation
is sufficient to achieve the best translation results, i.e. all land cover can be inferred from
a single image representation. Moreover, the late strategies appear to be the best suited
in the specific case of land-cover translation, with a shared representation space for all
land-covers. We thoroughly examined the effects of three different kinds of additional data
/optical, SAR, DEM) and observed that fusing additional data with a map representation
significantly improves the results, provided that the source map is not too coarsely resolved
compared to the target. Indeed, a coarsely resolved source map conveys little information
on the expected output class and does not bring extra information.

To sum up, this section presented significant improvement to the learnt spatial and
geographical context translation method by increasing the quality of the translation,
the spatial generalisation ability, and the noise robustness. However, the current multi-
landcover strategy appears widely inefficient as (i) multiple source classes with close or
identical content are learnt separately, e.g. the "coniferous forest" of the CLC, OSO and
OCSGEc map are all mapped using a separate encoder, (ii) multiple target classes with
close or identical content are learnt separately, e.g. the network learns separately the
concept that "herbaceous near water" is likely wetlands for the OSO to CLC translation
and the OCSGEu to CLC translation. This fosters multiple operational constraints: a
source map can only be translated into one of the target maps used during training and
only if the source map was also included in the training. For instance, the MLCT-Net we
trained on the six land cover is able to translate one of those maps into one of the other
five but can not use as a source or target any other maps even if it exhibits close semantic
and resolution characteristics. We address this problem in the following chapter.
# Building a semantically continuous land-cover representation

## 6.1 Introduction

Previous sections considered the translation as the transformation between a discrete representation space with n dimensions formed by the $c_S$ source classes to another with $c_T$ target classes. In those sections, the CGLS Closed Forest and OCSGE Broad-leaved trees translations are learnt distinctly using two different one-hot encodings (see Equation 6.1). As each nomenclature is considered an independent representation space, adding a new source or target nomenclature requires retraining the algorithm from scratch.

$$ 
\begin{aligned}
CGLS_{ClosedForest} &= [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] \\
OCSGEcover_{BroadLeavedTrees} &= [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0]
\end{aligned}
\qquad (6.1) $$

We observe that a person taught that CGLS Forest is translated either into OCSGE use Forestry or Without use, depending on the context, is able, without additional information, to suggest that MOS Forest can be translated similarly. This ability to generalise a translation model to new source and target nomenclatures without specific training is termed zero-shot translation.

Contrary to our methods, the person is aware of the semantic links between the different classes. She/He relies on a global understanding of class definitions to estimate a resemblance between classes. She/He assumes that a unknown class must be approximately translated to classes that resemble it the most amongst the ones he learned. We assimilate this understanding to a capacity to represent the translation not as the transformation from a discrete representation space to another but as a change in the interpretation of a single representation space including all possible land-cover labels.

Finding a way to train the model on such a common nomenclature representation space could avoid retraining the network with each new source or target nomenclature. This could be particularly useful when no preexisting training samples of the new nomenclature is available.
The approaches currently representing multiple nomenclatures in a unified space have been presented in Section 2.1.1.1 under the term *standardisation* (LCCS, EAGLE). However, those semantic spaces are unsuitable for our paradigm, as their representation spaces are discretely defined. For instance, EAGLE offers more than 570 dimensions (expandable), describing distinct attributes with seven discrete values according to how mandatory the attribute is/is not. The notion of distance between two classes is not clearly defined. For instance, the distance between classes for which the second attribute is either valued x (The attribute is not relevant for the class) or 0 (the attribute is not in the definition) or 2 (The attribute is mandatory) require defining subjective rules. Moreover, it requires weighting each attribute/dimension arbitrarily.

<figure><img src="image_60.png" /><figcaption>Figure 6.1: Manually defined 2D semantic representation space for the OCSGE cover and CGLS maps. The closer two classes definitions are, the closer their semantic encoding.</figcaption></figure>

We propose a method projecting a class into a continuous nomenclature representation space based on its definition. Like the person's mental representation space, it should bring two semantically close classes near in such a space. The metric space in which semantically similar classes are close to each other is called *semantic space* in the literature. Instead of one-hot encoding CGLS *Closed Forest* and OCSGE cover *Broad-leaved trees*, we encode them in a single space in which they are closer to one another than classes such as OCSGE cover *Bare soil*. We illustrate this idea by proposing a 2D manually defined semantic representation space with the classes of these maps in Figure 6.1. The resulting encoding of the two previous classes is then given in Equation 6.2.
$$ 
\begin{aligned}
CGLS_{ClosedForest} &= \begin{pmatrix} -0.6 \\ -1.2 \end{pmatrix} \\
OCSGEcover_{Broad-leaved} &= \begin{pmatrix} -0.45 \\ -1 \end{pmatrix}
\end{aligned}
\qquad (6.2) $$

This semantic encoding can replace the standard one-hot encoding as input and output data of the networks, as shown in Figure 6.2. To date, the few examples of the application of semantic spaces in land-cover mapping are mainly in zero-shot image classification or segmentation. A model is trained to project pixels into a semantic space obtained by applying a pre-trained language model [246]. Unlike image classification, in translation, the semantic representation space can be applied to both source data (a map) and target data (another map). Encoding the source maps provided to **MLCT-Net** during training could allow spatial context-wise translation of maps never seen during training at the time of inference (e.g. US NLCD → CLC) based on the idea that the translation of the unseen classes fallows approximately the same contextual rules than the classes seen during training with an encoding close to the unseen class. We refer to this scenario as *zero-shot source translation*. Encoding the target maps, could allow each source map object to be projected differently into the semantic space depending on its spatial context. For instance, a pixel of OCSGE *Herbaceous formations* could be projected closer to *Herbaceous wetlands* or *Herbaceous vegetation* depending on its distance from a water point. Instead of outputting a fixed nomenclature with *n* classes, we could obtain a continuous representation that can be derived into a nomenclature never seen during training (CLC → NCLD). We refer to this scenario as a *target zero-shot* solution.

Training models or comparing methods producing semantic representation space requires assessing the quality of the nomenclatures encoding on a ground truth. As no public dataset with multiple nomenclatures has been released yet, we created a small land-cover definition dataset (LCDD). Ten nomenclatures are selected for their semantic diversity: the 6 of the MLULC dataset, the one of ESRI land-cover map<fnref n="1" />, and three different nomenclature of the MODIS Land Cover Type MCD12Q1 [290] using different standardisation approach. We introduce the LCDD 169 class definitions in Appendix D. The following section discusses the expected properties desired for a semantic space and related evaluation criteria. We underline that the small size of the dataset prevents training complex models but can be used to train very simple models.

<footnote n="1">¹https://livingatlas.arcgis.com/landcover/</footnote>
<figure><img src="image_61.png" /><figcaption>Figure 6.2: Comparison between a translation with one-hot encoding using a **MLCT-Net** with a u-dimension shared representation space (top) and a d-dimension semantic encoding (bottom) for land-cover map A ($c_A$ classes) and B ($c_B$ classes). A similar colour represents a comparable representation space, *i.e.* same number of dimensions encoding comparable features. Semantic encoding requires a single network since both the input and output share the same representation space. **MaxDim:** classifies the final map as the class with the highest logit value. **ASPP:** atrous spatial pyramidal pooling.</figcaption></figure>

Our zero-translation framework based on semantic representation space can be decomposed into three main steps:

1. Class definitions are encoded using a language model in a continuous semantic representation space. Two approaches have been proposed in the literature so far: a bag-of-words [58] trainable using a few land-cover classes definitions such as those
of the LCDD and a Word2Vec [246]) model trained on large generic text corpora
not land-cover specific. We compare those two approaches and propose a third one
based on a transformer architecture trained generic text corpus.

2. The three explored techniques result in very high dimensional representation spaces (300 - 1 000 dimensions) that are unusable to encode land-cover data fed to a network for memory concerns. Moreover, models trained on generic text corpora encode for semantic notions irrelevant from a land-cover point of view e.g. the word water is close to surf. Different supervised or unsupervised dimension reduction techniques are explored to reduce the dimensions. In particular, we introduce a new supervised dimension reduction technique based on a multi-layer perception trained using the small LCDD dataset.

3. Land-cover maps used to train a translation CNN are first semantically encoded using the dimension-reduced SRS. At inference, an unseen map is encoded using the same semantic space and fed to the network.

The first section defines a set of desired properties for a land-cover semantic representation space and related evaluation criteria. We point out that this first section is part of our original contributions, as the only two previously existing works [58, 246] did not define them explicitly. The second section introduces and compares the different natural language models and dimension reduction techniques based on the evaluation criteria presented in the first section. Finally, we propose the first example of applications of these semantic spaces to the zero-shot translation case.

## 6.2 What is an ideal semantic representation space ?

A **Semantic Representation Space (SRS)** aims to reflect the proximity between different concepts in a space with a finite number of dimensions [195]. In our land-cover case, we aim to obtain a continuous metric space endowed with a distance metric (Euclidean distance, for example) in which semantically close classes are also close in the sense of the metric. This section first defines the notion of land-cover SRS in a broad way independently from our desired application (zero shot translation). In particular, we discuss, how and what to encode in class definitions and the notion of proximity in the SRS and its link semantic similarity. We then define additional characteristics that the space should exhibit to be usable in the specific case of zero-shot translation.

For simplification, we use the notations of Section 2.1 (see Figure 6.3). *S*ᵢ and *T*ⱼ are sets holding all the source pixels classified *i* (out of *c*<sub>S</sub> classes) and all the target pixels labelled *j* (out of *c*<sub>T</sub> classes). *s*ᵢ, *t*ⱼ denote the descriptors of the source class *i* and target class *j* in a considered standardisation system. *s*ᵢ(*v*) denotes the v<sup>th</sup> descriptor of the source
<figure><img src="image_62.png" /><figcaption>Figure 6.3: Illustration of encoding of classes using a standardisation system (EAGLE) and a continuous semantic representation space. Standardisation systems use discrete variables (x: "should not be included", 0: "can be included but not in definition ..."), while SRS use abstract continuous variables (here 5). For clarity only 15 discrete variables are displayed for EAGLE while the matrix used for CLC includes more than 350 variables.</figcaption></figure>

class i in the standardisation system. $s'_i$, $t'_j$ denote their descriptors in the SRS, with $s'_i(u)$ being the $u^{th}$ dimension out of $q$ dimensions of the SRS. We underline that $s_i(v)$ might be a discrete encode variable, depending on the standardisation system, while $s'_i(u)$ is a continuous variable from the SRS.

## 6.2.1 Encoding class definitions

Our semantic space should be able to encode for all information present in the class definition. The space must include semantic properties of the objects, including their cover and use. Additionally, it should integrate spatial resolution information (commonly provided in the form of a threshold). Lastly, it should differentiate between inclusion and exclusion statements, e.g. if *Crop* definition excludes pastures, *Crop* should not be close to *Pastures* even though the term pasture appears in its definition.

In the particular case of hierarchical nomenclature, we consider each hierarchical level as a distinct nomenclature, accumulating the semantic content of the previous levels. We do not aim to create multiple sub-spaces encoding the notion of class hierarchy separately, as the hierarchical division of nomenclatures is often widely different from one map to another.

As underlined before, the SRS is used to encode land-cover maps before feeding them two Convolution neural networks such as those presented in the previous sections. As such, the definitions should be encoded with continuous variables that enables straightforward distance computation or interpolation. We insist that standardisation approaches such as LCCS or EAGLE result in high-dimensional discrete spaces encoding with discrete variables. In the remainder, we assume a continuous space, thereby neglecting the standardisation approaches.
## 6.2.2 Computing proximity between classes

Class SRS encoding must reflect the semantic proximity between class definitions. Analysing the quality of a semantic space requires: (i) defining a measure $M_1$ of proximity between class definitions and a measure $M_2$ of proximity between classes in SRS, and (ii) defining the correlation link expected between M1 and M2.

### 6.2.2.1 M1: Proximity between classes

Computing the proximity between class definitions can be achieved using three different measurement methods.

The **Binary Overlap Measure (BOM)** is based on a manual distinction between classes sharing elements in common with others. This binary metric is implicitly used by nomenclature translation system relying on expert knowledge [1]. As the binary behaviour does not allow a satisfactory comparison of the degree of resemblance, a few more evolve techniques such as [55, 56] assess it using a three level granularity (expected, uncertain, or unexpected). As there is little doubt about the existence or not of an element in common between to two classes, BOM is rarely false.

$$BOM(s_i, t_j) = \begin{cases} 1 & \text{if } S_i \cap T_j \neq \emptyset, \\ 0 & \text{else.} \end{cases} \qquad (6.3)$$

Section 2.1.1.2 introduced the notion of **Semantic Similarity Measure (SSM)** (Equation 2.8). Often based on a proportion of attributes shared between two classes, it is adopted by most semantic approaches. It measures the resemblance with a higher granularity than BOM. However, it is often very biased because it depends enormously on the relative weight given to each attribute, arbitrarily fixed in the method.

The **Statistical Co-Occurrence Measure (SCOM)** computes the proportion of pixels of each target class corresponding to a given source. This measurement is normalised by the proportion of each target class. It offers a high granularity while avoiding arbitrary weights. Source/target map errors might reduce the correlation between this observed statistic relation and the expected semantic one. Conversely to SSM and BOM, SCOM is only based on measurement conducted on real map samples and ignores class definitions.

### 6.2.2.2 M2: Measuring proximity in the representation space

We restrict this manuscript scope to SRS in Euclidean geometry. We underline that approaches encoding in non-Euclidean geometries, such as hyperbolic spaces, have shown encouraging results, especially for encoding hierarchical data in low-dimensional spaces [162, 227]. They require a very high numerical precision for distance computation (64 or even 128-bit): we consider them unsuitable and deliberately ignore them.
In Euclidean space, the Euclidean distance is commonly adopted for distances (Equation 6.4). Many works have pointed out that this measure is unsuitable for high-dimensional spaces, which may be problematic for some of our semantic spaces.

$$L_2(s'_i, t'_j) = \sqrt{\sum_{u=1}^{q} (s'_i(u) - t'_j(u))^2}. \qquad (6.4)$$

The Manhattan distance is often used as a replacement for higher-dimensional spaces (see Equation 6.5). As the Euclidean distance, the Manhattan one is not scale-invariant. This is not an issue as long as the amplitude of variation of each dimension correlates to the semantic gap.

$$L_1(s'_i, t'_j) = \sum_{u=1}^{q} |s'_i(u) - t'_j(u)|. \qquad (6.5)$$

In other cases, a commonly solution in high dimensional spaces is the cosine distance (Equation 6.6). Instead of focusing on the difference $s'_i$ and $t'_j$, the cosine distance computes the cosine of the angle between both. CosineDistance = 0 when vectors are collinear, 1 when they are orthogonal, and 2 when they are opposite.

$$\text{CosineDistance}(s'_i, t'_j) = 1 - \frac{\sum_{u=1}^{q} s'_i(u) t'_j(u)}{\sqrt{\sum_{u=1}^{q} (s'_i(u))^2} \sqrt{\sum_{u=1}^{q} (t'_j(u))^2}}. \qquad (6.6)$$

### 6.2.2.3 Relating M1 and M2 scores

Let $M_1(s_i, t_j)$ be the measure of the proximity between $s_i$ and $t_j$ definitions. We denote $M_2(s'_i, s'_j)$ the proximity of their respective encodings. $C$ is the constraint the semantic space should minimize. We propose the distinction between two kinds of semantic spaces.

**Distance-conservative semantic spaces (DCSS)** aim to ensure the distance between two encodings is identical to one of the continuous proximity measurements (SSM, SCOM, Equation 6.7). Training a natural language model or a dimension reduction model to obtain a DCSS is a difficult task as it requires a training dataset associating to each couple of source target class an expected distance value. No such dataset currently exist for significant number of maps and would be difficult to define as current SSM and SCOM measures does not ensure the preservation of the triangular inequality: some inter-class distance might be mutually exclusive.

$$C_{\text{DNCSS}}(s'_i, t'_j) = |M_1(s_i, t_j) - M_2(s'_i, t'_j)|. \qquad (6.7)$$
**Neighbour-conservative semantic spaces (NCSS)** aim to ensure that classes with close definitions are encoded more closely in the semantic space than classes with far definitions. NCSS only constraints the neighbourhood without constraining the distance: the distance between *Water* and *Cereals* can be 0.2 or 3000 as long as it is higher than the distance with *Maize*. This makes models targeting NCSS easy to optimize even for NCSS with low number of dimensions. We distinguish two subcategories of NCSS, Absolute-Neighbour conservative semantic spaces (ANCSS) and Relative-Neighbour conservative semantic spaces (RNCSS). For ANCSS, target classes partially corresponding to a given source class (BOM = 1) are closer to this class in the semantic space than any other target class (Equation 6.8). They distinguish close/far classes in a binary manner. For instance, *Cereals*, *Maize*, and *Tubers* should be closer to the source class *Agriculture* than any other, but there is no constraint on how far each class should be or if *Maize* is closer than *Cereals*. Conversely, RNCSS (see Equation 6.9) ensures that each target class encoding respects the correct proximity ordering to a source class by defining each neighbour relatively. For instance, *Maize* should be closer to *Agriculture* than *Cereals* which should be closer to *Agriculture* than *Water*.

$$C_{ANCSS}(s'_{i}, t'_{j}) = \begin{cases} M_{2}(s'_{i}, t'_{j}) & \text{if } BOM(s_{i}, t_{j}) = 0. \\ 0 & \text{else.} \end{cases} \qquad (6.8)$$

$$C_{RNCSS}(s'_{i}, t'_{j}, t'_{k}) = \begin{cases} |M_{2}(s'_{i}, t'_{j}) - M_{2}(s'_{i}, t'_{k})| & \text{if } M_{2}(s'_{i}, t'_{j}) > M_{2}(s'_{i}, t'_{k}) \text{ & } M_{1}(s_{i}, t_{j}) < M_{1}(s_{i}, t_{k}). \\ |M_{2}(s'_{i}, t'_{j}) - M_{2}(s'_{i}, t'_{k})| & \text{if } M_{2}(s'_{i}, t'_{j}) < M_{2}(s'_{i}, t'_{k}) \text{ & } M_{1}(s_{i}, t_{j}) > M_{1}(s_{i}, t_{k}). \\ 0 & \text{else.} \end{cases} \qquad (6.9)$$

For land-cover translation, an ANCSS is sufficient as translating an unknown source nomenclature can be achieved by simply translating the unknown class similarly to the few closest classes in the SRS. As training, a model to produce an ANCSS is significantly simpler (does not require knowing the precise semantic distances, nor the relative ones), this manuscript only considers ANCSS.

### 6.2.2.4 Evaluation of the SRS

We evaluate the SRS on its ability to ensure that classes determined as a neighbour in the LCDD dataset (BOM=1) are also neighbours in the SRS. We first describe how we determined neighbour classes in the LCDD dataset and then introduce two quantitative metrics assessing the neighbour preservation quality.

**Determining neighbours classes in the LCDD dataset** Neighbourhood inside the LCDD dataset (BOM) is determined by hand independently for each source/target couple of nomenclature. Following notations of Section 2.1, we consider that $S_i$ has a single neighbour $T_j$ when they include the same objects (Equation 2.3) or when all the objects
of $S_j$ are included in $T_j$ (Equation 2.4). When $T_j$ is included in $S_i$ (Equation 2.5), $S_i$ has multiple neighbours, all classes included in $S_i$. Without overlap (Equation 2.6), the classes are considered as non-neighbours. As mentioned in Section 2.1, determining by hand the pairs of classes establishing these relationships is relatively simple and leads most of the time in consensus results between several analysts [339].

The distinction of neighbour classes is more difficult to apprehend when a source class partially overlaps with several target classes [57] (see Equation 2.7). Generally, we consider here that all the target classes establishing a partial overlap are neighbours. However, we place two notable exceptions. When the partial overlap is due to a difference in the spatial resolution of the analysis, $S_i$ may have the constraint of necessarily including a certain number of target classes ($T_j, T_k...$) without the converse being true. For instance, road and airport are partially overlapping as some road pixels do not belong to airports, and some airport pixels do not belong to roads. However, an airport necessarily contains roads (the runway), while the reverse statement is invalid. In this case, we consider an asymmetric link in which $S_i$ (airport) is close to $T_j$ (road) without the converse being true. The second exception is when the partial overlap is due to an apparent lack of correspondence between a source class and the set of target classes. The CLC *Wetland* class has no equivalent in the OSO, MOS, or OCSGE nomenclatures. Then, an asymmetric neighbour link is also used (*Wetland* is considered a neighbour with *Water* or *Herbaceous areas* in the OCSGE, while the converse is invalid).

We point out that determining a neighbourhood relationship between a source class and the classes of several nomenclatures simultaneously is not considered. This would imply some difficult determination such as the *Forest* of map A is closer to the *Forest* of map B than the *Forest* of map C. Consequently MOS *Water* necessarily has at least five neighbours among the MLULC nomenclature (one per land-cover map). Using the above protocol, we manually define the neighbors of the 169 classes of the LCDD independently for each target nomenclature (Appendix B).

**Evaluation of SRS neighborhood preservation** We derive two distinct neighbour conservation quality metrics (Figure 6.4). First, we define the **Closest Neighbour** metric (CN) as the proportion of source class encoding having for closest target encoding one of the expected neighbour (BOM=1). We simply average Equation 6.10 for all source and target couples. In practice, this metric directly reflects the proportion of classes for which the closest neighbour in the SRS is one of the possible neighbours.

$$ CN(S, T) = \frac{1}{c_S c_T} \sum_{i=1}^{c_S} \sum_{j=1}^{c_T} f(S_i, T_j) \\ \text{with } f(S_i, T_j) = \begin{cases} 1 & \text{if } \arg\min_{t'} M_2(s_i', t') = t_j' \text{ and } BOM(s_i, t_j) = 1 \\ 0 & \text{else} \end{cases} \qquad (6.10) $$

Secondly, the **Neighbor preservation (NP)** metric evaluates the overall preservation of
all neighbours. The proportion of target encoding labeled as not neighbouring the encoded
source class but closer to it than at least one of the expected neighbours. For instance, if
the representation of *Cereals* has for nearest neighbour *Corn*, then *Water* and *Rapeseed*,
whereas *Corn* must be the neighbour of *Corn* and *Rapeseed*, this measurement equals 2/3.
In practice, a source class with *x* neighbours in the target nomenclature achieve a 1 NP
when the *x* closest element of the source class in the SRS are only those neighbours. A
measure close to 0 indicates that multiple non-expected neighbouring classes are closer
than one of the expected neighbours.

<figure><img src="image_63.png" /><figcaption>Figure 6.4: Illustration of the Closest Neighbor (CN) and Neighbor preservation metrics (NP) considering a single source class example. The metrics over a nomenclature are computed by the averaging the results over all source classes.</figcaption></figure>

6.2.3 Constraints specific to the zero-shot use-case

This section identifies additional criteria when the SRS is aimed specifically to be used to
encode map feed to a CNN to achieve zero-shot translation.

First, a model trained to project n nomenclatures in an SRS should be able to encode a
new nomenclature unseen during training without changing the encoding of the n first
nomenclatures. Machine learning-based models that project class definition into the SRS
should thus be trained inductively and not transductively.

A second immediate consequence is that it must guarantee that the encoding of two
different nomenclatures is not spatially disjoint. As nomenclatures have very different
description schemes, the model is likely to encode different nomenclatures at different
locations in the SRS space. This representation shift could prevent the network from
interpolating the translation of the class of an unseen nomenclature based on the position of
the seen one, as it could be encoded in a completely different way. We propose two simple
unsupervised shift measurements, **Inertia** and **CH**, that aim to quantify the average shift
between two nomenclature representations.
<figure><img src="image_64.png" /><figcaption>Figure 6.5: Illustration of the Inertia and Cluster homogeneity (CH) metrics (three nomenclatures). *Inertia* compares the average variation of distance of the nomenclatures to the SRS. *CH* evaluates the homogeneity of the nomenclature distribution into the SRS space. The desired SRS should look like the one in the first column.</figcaption></figure>

Under a no-shift between nomenclature scenario, all nomenclatures should approximately have the same centroid coordinates as they are on average at the same location in the SRS (as illustrated in the first row, the first column of Figure 6.5). The inter-nomenclature **Inertia** measurement gives an order of magnitude of the dispersion around the empirical centre of SRS. It compares the average distance between the centroid of the SRS (centroid of all classes) and the classes of a given nomenclature with the average distance between the centroid and the classes of all other nomenclatures. We normalise this value in order to get a distance percentage of variation. For instance, a nomenclature A with Inertia of 0.1 indicates that the classes of A are, in average, 10% further from the centroid of all classes than the classes of other nomenclatures. Let *V* denote the centroid of all the classes independently from their nomenclature, and *N* denote the set of all the *v* nomenclatures available. We denote *N*<sub>*h*</sub> the h<sup>th</sup> nomenclature, *N*<sub>*h*<sub>i</sub></sub> the i<sup>th</sup> of the h<sup>th</sup> nomenclature, and *c*<sub>*h*</sub> the number of classes of *N*<sub>*h*</sub>. Inertia is given by Equation 6.11. This metric is directly inspired by the notion of intra-class and inter-class inertia used by the clustering algorithm, which aims either to minimise intra-class inertia or to maximise inter-class inertia (achieving one objective achieves the other). Our inter-nomenclature inertia measure is equivalent to
the notion of inter-class inertia, but should be minimised rather than maximised.

$$ \text{Inertia}(N_h) = \frac{\frac{1}{c_h} \sum_{i=1}^{c_h} M_2(N_{h_i}, V)}{\frac{1}{v} \sum_{\substack{j=1 \\ j \neq h}}^{v} \frac{1}{c_h} \sum_{i=1}^{c_h} M_2(N_{j_i}, V)} \quad (6.11) $$

The second, based on the hierarchical clustering algorithm [222], aims to verify that the classes of each nomenclature are approximately homogeneously dispatched in the SRS. The hierarchical clustering algorithm is tasked to find the same number of clusters as the number of nomenclatures. In a homogeneous space, each cluster should include approximately the same proportion of each nomenclature as the proportion observed globally e.g.. If OSO represents 10% of the classes, each cluster should include approximately 10% of OSO classes. This shift measurement evaluates the distance to this homogeneity objective.

Let $P(N_h) = \frac{c_h}{\sum_{j=1}^{v} c_j}$ represent the proportion of classes belonging to the h nomenclature.

Let $AG_i$ denote the i-th cluster, $AG_i(N_h)$ denote the number of classes belonging to h nomenclatures in the i-th cluster. The cluster homogeneity value CH (Equation 6.12) is only used to compare different SRS: there is no theoretical obligation that each cluster exhibits precisely the same proportion of each nomenclature. However, it can be used to compare different SRS with comparable CN or NP measurement: a lower cluster value indicates a more homogeneous space in which each nomenclatures is homogeneously dispatched in the space.

$$ CH(N_h) = \sum_{i=1}^{v} M_2\left(\frac{AG_i(N_h)}{\sum_{j=1}^{v} AG_i(N_h)}, P(N_h)\right) \quad (6.12) $$

The **Neighbor Stability (NS)** evaluates the potential to interpolate encoding between two neighbouring classes while preserving neighbourhood links (Figure 6.6). Let $s'_i$ and $t'_j$ be two neighbouring classes. Any encoding between $s'_i$ and $t'_j$ should be closer to $s'_i$, $t'_j$ or one of their neighbours than any non-neighbour classes. For instance, an encoding between *Cropland* and *Wheat* should be closer to *Corn* than *Water*. To verify this characteristic, we regularly sampled 20 points on a segment going from the source to the target class. We estimate the proportion of the segment closer to one of those neighbours by computing the proportion of those 20 points closer to one of the potential neighbours than a non-neighbouring element. A 100% NS implies that approximately 100% of the segment between two neighbouring elements respects the neighbour's preservation. In those conditions, one could imagine using simple arithmetic combinations between classes while ensuring that the results are still be semantically meaningful e.g. $\frac{Water+Crops}{2}$ should be closer to *Rice* than *Forest* when NS is high.

Lastly, we point out that an SRS need to have a small number of dimensions (less than
100) to avoid excessive memory consumption in perspective to encode land-cover map feed to CNN. To give an order of magnitude of the problem, a 600×600 pixel patch encoded in a 100-dimensional space using a float 32 precision represent a 1.15 GB memory usage alone. The 32GB GPU used in this thesis would only be able to load 26 patches at a time, decomposed in 13 input and 13 output, without even considering the memory consumption of the network computation itself. From a theoretical point of view, small dimensional SRS are sufficient to encode fairly the notion of proximity between classes, as evidenced by the encoding in only two dimensions proposed for two nomenclatures in Figure 6.1.

<figure><img src="image_65.png" /><figcaption>Figure 6.6: Neighbour Stability (NS) metric illustration between one source and one target class. The considered target class is one of the expected neighbors of the source. NS estimates the percentage of elements on a segment from source to target that are closer to one of the expected neighbours of source ($BOM(s_i, t_j) = 1$) in the target nomenclature NS$_{source}$ than any other target class. Reciprocally NS$_{target}$ estimates the elements that are closer to one of the expected neighbor of the target class in the source nomenclature ($BOM(t_i, s_j) = 1$) than any other source class.</figcaption></figure>

## 6.3 How to build a SRS for land-cover translation ?

The main difficulty faced when aiming to obtain a model projecting land-cover classes in a continuous semantic representation space is the lack of wide land-cover definitions dataset preventing to train task-specific complex language model. To alleviate this issue we propose a two step procedure. First we project the land-cover definition into an SRS by using either a simple language model that can be trained on our small LCDD dataset or more complex models trained using generic text corpora unrelated to land-cover. We evaluate those models based on their capacity to respect the properties mentioned in the previous section. The different obtained SRS exhibits the two same limitations: (i) some
of the abstract variables used to encode classes takes into account concepts irrelevant from a land-cover point of view (ii) the SRS are to high dimensional to be usable to encode maps fed to a convolution neural network (>300 dimensions). Therefore, the second step compares different dimension reduction techniques aiming to recombine the numerous encoding variables into a smaller number of well designed one. In particular we explore a supervised dimension reduction technique trained to optimise the neighbourhood constraints defined in the previous section using the LCDD dataset. In the remainder, we denote HDSRS, the high-dimensional SRS obtained as the output of one of the three previous methods and LDSRS, the low-dimensional SRS we aim to obtain.

## 6.3.1 Using language models to obtain high-dimensional SRS

As only a few works have been conducted in obtaining SRS from land-cover definitions, this section evaluates the current main semantic embedding models. We first present a Bag-of-Word model proposed for land-cover translation by [58] that we train directly on the LCDD dataset. Then we present a Word2Vec model proposed by [246] for zero-shot land-cover semantic segmentation from images trained on large text corpora. Lastly, we propose replacing those two models (the only one found in the literature for land-cover class embedding) with a transformer-based architecture trained on text corpora that we believe more adapted to work the complex sentences of class definitions.

### 6.3.1.1 Bag-of-words

**Overall idea** The bag-of-word approach [112] (BOW) proposed by [58] is the only model applied to the land-cover translation task in the literature. This model determines all the words used in the land-cover definitions to build a land-cover term dictionary. Each definition is then encoded as the count of the words inside it. For instance, in a setup where the only available land cover definitions are (5) and (6), the dictionary is {"Forest", "natural", "tree", "vegetation", "with", "higher", "than", "5m", "Orchards", "Cultivated"}.

"Forest : natural tree vegetation with tree higher than 5m" (5)

"Orchards: cultivated tree " (6)

Using each word of the full dictionary as one of the encoding dimensions of the BoW with a value equal to the number of word occurrences in the definition, we obtain Figure 6.7.

<table><caption>Figure 6.7: Bag-of-word encoding of two schematic classes Forest (5) and Orchards (6)</caption><thead><tr><th></th><th>Forest</th><th>natural</th><th>tree</th><th>vegetation</th><th>with</th><th>higher</th><th>than</th><th>5m</th><th>Orchards</th><th>Cultivated</th></tr></thead><tbody><tr><th>Forest</th><td>1</td><td>1</td><td>2</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0</td><td>0</td></tr><tr><th>Orchards</th><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1</td></tr></tbody></table>
In practice, this encoding is very sensible to the class definition length thus [58] normalise the encoding values by the number of words in the class definition e.g. Orchard encoding becomes *Orchards* = [0, 0, 1/3, 0, 0, 0, 0, 0, 1/3, 1/3]. In order to emphasise the importance of class-specific words, they adopted a common total frequency times inverse document frequency weighting scheme. Each term is weighted such that a term that frequently appears in one definition but not in the other exhibits a high weight. Let $n_{s_i}(t)$ be the number of times the word $t$ appears in the definition of class $i$, $L_{s_i}$ be the number of words in class $i$ definition, $h(t)$ be the number of classes including $t$ in their definitions and $D$ be the total number of classes. The resulting weight is obtained using Equation 6.13.

$$s'_{i}(t) = \frac{n_{s_i}(t)}{L_{s_i}} \ln \frac{D}{h(t)} \qquad (6.13)$$

**Implementation details** We follow [58]. Each class description is first converted into a list of terms that are used to build the dictionary. In most cases, a term represents a single word but it also includes small phrases for distances/surfaces informations (e.g. "5 m"), and locations (e.g. "Alpine grasslands"). Unlike [58], we add a few more constraints due to the small number of definitions and their limited size: each word is considered case insensitive, and plural and singular words are considered as a single key. This results in a 987-term dictionary inducing a 987-dimension HDSRS.

**Strengths and weaknesses of the method** The core advantage is that the model is directly built on existing land-cover definitions. The HDSRS only considers concepts related to land cover and ignores unrelated concepts. However, this approach suffers multiple limitations:

* the method is entirely dependent on the exhaustivity of the dictionary. As we considered that the HDSRS should be obtained inductively to enable zero shot land-cover translation, a nomenclature not used to build the HDSRS can include terms not encodable using the original dictionary. This results in a partial loss of class information.

* the method ignores grammar and order. This prevents learning concepts such as inclusion and exclusion. For instance, a statement such as "this class includes A and excludes B" is encoded as the reverse statement.

* the distance is based on the weighted co-occurrence of words between the definitions. It does not consider word meaning or semantic importance. Classes "Forest: high trees" and "Conifers: needle-leaved stands" have no common words but describe the same object. Consequently, the notion of threshold is also not encoded. For instance, a Forest defined as "with at least 80% coverage" is not closer from one with a 70% threshold than one with a 10%.
### 6.3.1.2 Word2Vec

**Overall idea** Instead of producing an encoding per class based on word frequencies like the BOW, the Word2Vec approach [215], aims to provide an encoding per word taking into account semantic meaning. The core idea is that a word semantic meaning can be inferred by analysing the words frequently used next to it. For instance, *Coniferous* is often located close to the word *tree* in a sentence, semantically linking those two terms. Word2Vec have been explored to perform many zero-shot classification tasks, especially semantic segmentation [32]. To our knowledge, the only attempt in the land-cover field deals with remote-sensed image zero-shot semantic segmentation [246]. This work did not focus on translation and encoded solely the class name and not the class definition, making it hardly generalisable to translation: many classes can have the same name (thus the same encoding) but different content.

The word embedding procedure consists in building a dictionary of all possible words leading to a one-hot encoded version of each word (the same 10-word dictionary as above results in $tree = [0, 0, 1, 0, 0, 0, 0, 0, 0, 0]$). A small 1-hidden layer perceptron processes sentences with a sliding window trying to predict the $n^{th}$ word given the x previous and x later words. For instance, in the previous example, using $x = 2$, one iteration of the algorithm on "Forest : natural tree vegetation with trees higher than 5m" could consist in training the perceptron to predict *vegetation* given the 2 previous (*natural*, *tree*) and 2 later words (*with*, *tree*) (Figure 6.8). The first layer of the perceptron processes each of the four one-hot encodings iteratively in a d-dimensional space (using d neurons). The four d-dimensional vectors are averaged and fed to the second linear-layer, predicting the middle word.

<figure><img src="image_66.png" /><figcaption>Figure 6.8: Illustration of the Word2Vec algorithm using a window of size 2. Once trained, the word embedding is the output of the first linear layer (orange arrows).</figcaption></figure>

Once trained on several sentences, the word embedding is then considered to be the $d$-dimensional space obtained after the first layer. We obtain the class definition encoding
as the average of all its words encoding.

**Implementation details** We used the Word2Vec implementation available at². The model is pre-trained on the Google News datasets, including more than 300 billion words. As for the BoW method, we observe that some word combinations are essential. However, instead of defining manually a set of rules of which words should be gathered, the Word2Vec developers identify terms that are frequently similarly ordered based on n-gram models [214]. The pre-trained Word2Vec model uses a 10-dimensional window and encoded in a 300-dimensional HDSRS.

**Strengths and weaknesses** This method partially alleviates the BoW limitations by replacing definition encoding based on terms frequency with an average of estimated word semantic meaning. The computed distance in the semantic space conveys more semantic meaning than the BoW approach. However, Word2Vec ability to reflect semantic meaning widely depends on the window size, the number of words used for training and *d*. The small size of the LCDD dataset prevents from training the Word2Vec method: a pre-trained Word2Vec is thus used. Some of the resulting encoding dimensions might be irrelevant for land cover as the pretraining is conducted on a generic dataset. For instance the closest term embedding of *Coniferous* is *In your garden* and the second closest of Ocean is *Neptune*. Additionally, this method still disregards word order as the class encoded is realised by averaging all the words encoding of the definition independently from their order.

### 6.3.1.3 Transformer based encoding

**Overall idea** Alleviating the limitations of the previous methods requires: (i) developing a method to address words outside the dictionary encountered at inference, (ii) taking into account word ordering, (iii) proposing a context-wise encoding (e.g. the terms *water* in the definition of *rice*, *wetland*, *water course*, and *sea* have close yet not identical meaning). In 2018, BERT was introduced in [66] to tackle all those issues simultaneously by combining several independent works handling each of those problems. Since various models have been proposed to improve the results but are based on the same overall idea. For clarity, we introduce concepts based on the original Bert implementation. We then discuss the difference with the recent model we used.

First they rely on tokenisation, i.e. sub-words units. The most straightforward tokenisation algorithm is to work directly on a per-character basis [50]. The dictionary includes each character of the training set individually. Per-character dictionaries are often complete even when using a small training dataset, e.g. considering only letters (26 characters). They do not suffer from out-of-dictionary issues at inference. However, each element of the dictionary conveys a very high diversity of semantic meaning depending on its context, making per-character tokenisation hardly usable. For instance, the letter *t* in the word

²https://radimrehurek.com/gensim/models/word2vec.html
tree or water has a completely different meaning. Various strategies proposing an trade-off
between word and character level tokenisation have been proposed based on sub-word
decomposition, water = wa + ter for instance. The BERT and subsequent methods are
mostly based on the sub-word tokenisation algorithm proposed by [331].

Secondly, to consider word ordering and achieve context-wide encoding, the BERT paper adopts the transformer architecture proposed by [318] that incorporates the positional encoding module. The key contribution of the BERT is to propose a way to consider the word context in a non-directional manner. They replace the traditional natural language processing objective of predicting the next word given the sequence of the previous ones with the task of predicting randomly masked words on the whole sentence. We underline that those architectures jointly produce an encoding per-word context-wisely (the same word is encoded differently depending on the sentence) as well as a sentence encoding. Hereafter, we consider that a class definition encoding is the average of all the word encodings.

The transformer architecture often comes with a hundred million parameters (approxi-
mately 300 million for BERT's original implementation) making them untrainable on the
tiny LCDD dataset due to overfitting concerns. Thus we rely on an available pre-trained
model. We underline that comparing the available models is difficult as most of them are
assessed on different tasks and datasets.

**Implementation details** As comparing all the available pre-trained models is unfeasible, we rely on a comparison of different top-performing models on 20 different datasets conducted by [251] and regularly updated<fnref n="3" />. We choose the MINILM [325] model, whose training uses two networks, a teacher (a regular BERT), and a student (a small BERT), enforcing the student to have the same attention parameters as the teacher. This distillation-based principle [122] enables a small student network to achieve comparable results to the big one. We rely on the student network. The implementation used is the official implementation provided by the hugging face library <fnref n="4" /> trained on a 1Billion word dataset <fnref n="5" />. This results in a 387-dimension HDSRS.

**Strengths and weaknesses** Even though this strategy alleviates most of the limitations encountered by the previous methods, the unavailability of a large land-cover-oriented dataset prevents from training on pure land-cover-oriented tasks: the distances computed in the semantic space still consider some irrelevant concepts.

<footnote n="3">³https://www.sbert.net/docs/pretrained_models.html</footnote>

<footnote n="4">⁴https://huggingface.co/microsoft/MiniLM-L12-H384-uncased?text=I+like+you.+I+love+you</footnote>

<footnote n="5">⁵https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2</footnote>
### 6.3.1.4 Comparison on the LCDD dataset

**Quantitative assessment** Unlike the two pretrained models (Word2Vec and MiniLM), the BoW is built using the LCDD dataset. To simulate the operational setup of a zero-shot land-cover translation, we iteratively use nine nomenclatures of the LCDD to train the model and use the last remaining one (termed *unseen nomenclature*) at inference. As some words of the unseen nomenclature can be out of the BoW dictionary, each model is different.

We compute the metrics mentioned in Section 6.2 for each model considering the unseen nomenclature as the source or the target. For instance, a BoW model trained using all nomenclatures except CLC, is evaluated using the metrics considering CLC either as the source or target nomenclature ($CN_{source}$, $CN_{target}$). We resume the results of those metrics using different $M_2$ measures in Table 6.1. Results from Inertia and CH values describe different notions depending on the $M_2$ measures. An inertia of 0.12 computed using Euclidean distance implies that in average the unseen nomenclature exhibits a 12% difference in terms of distance to the HDSRS centroid compared to seen nomenclatures. A 0.12 inertia computed using cosine distance implies a 12% angular variation. Thus, the best Inertia and CH values are compared independently per metric in this case. Conversely CN, NP and NS which are percentage based on $M_2$ metrics are comparable even when using different metrics thus we jointly assessed their best value for all metrics.

First, as the $M_2$ measure maximising the CN, NP and NS is almost exclusively the Cosine Distance, we consider it the most relevant metric. We underline that this was expected as the three spaces are high dimensional (from 300 to 987 dimensions) and some have been explicitly optimised for this metric (Word2Vec). Thus all the following observations are conducted by considering the results obtained with cosine distances.

A second observation is that the transformer-based encoding better preserves the closest neighbour (CN), the nearest neighbours (NP), and the overall shape (NS) at the cost of a worst inertia and CH value than the BoW approach. Therefore the transformer produces a better HDSRS from a pure proximity preserving point of view but introduces a slight semantic shift between land-cover maps. As the difference in CN and NN is very high (10 to 15% higher), we still consider Transformers as the most suitable solution.

<table><caption>Table 6.1: Quantitative of the three HDSRS obtained by three different semantic embedding methods</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">BOW</th><th colspan="3">PreWord2Vec</th><th colspan="3">MiniLM</th></tr><tr><th>euclidean</th><th>manhattan</th><th>cosine</th><th>euclidean</th><th>manhattan</th><th>cosine</th><th>euclidean</th><th>manhattan</th><th>cosine</th></tr></thead><tbody><tr><td rowspan="2">CN</td><td>source</td><td>39</td><td>37</td><td>60</td><td>58</td><td>59</td><td>64</td><td>60</td><td>59</td><td><strong>76</strong></td></tr><tr><td>target</td><td>32</td><td>37</td><td>58</td><td>58</td><td>59</td><td>64</td><td>60</td><td>59</td><td><strong>76</strong></td></tr><tr><td rowspan="2">NP</td><td>source</td><td>46</td><td>47</td><td>63</td><td>62</td><td>61</td><td>67</td><td>62</td><td>62</td><td><strong>76</strong></td></tr><tr><td>target</td><td>43</td><td>47</td><td>61</td><td>62</td><td>61</td><td>67</td><td>62</td><td>62</td><td><strong>76</strong></td></tr><tr><td rowspan="2">NS</td><td>source</td><td>83</td><td>91</td><td><strong>96</strong></td><td>88</td><td>88</td><td>92</td><td>90</td><td>90</td><td>95</td></tr><tr><td>target</td><td>59</td><td>64</td><td>68</td><td>65</td><td>65</td><td>67</td><td>66</td><td>66</td><td><strong>70</strong></td></tr><tr><td>Inertia</td><td></td><td><strong>0.13</strong></td><td><strong>0.05</strong></td><td><strong>0.07</strong></td><td>0.16</td><td>0.16</td><td>0.15</td><td>0.14</td><td>0.14</td><td>0.18</td></tr><tr><td>CH</td><td></td><td>0.4</td><td><strong>0.13</strong></td><td><strong>0.12</strong></td><td>0.31</td><td>0.38</td><td>0.278</td><td><strong>0.21</strong></td><td>0.47</td><td>0.17</td></tr></tbody></table>
<figure><img src="image_67.png" /><figcaption>Figure 6.9: PCA representation of the HDSRS obtained with a trained on generic text corpora Transformer.</figcaption></figure>

**Qualitative assessment**

Two different dimension reduction techniques are used to evaluate qualitatively those HDSRS. PCA is used to evaluate the global aspect of the HDSRS as it preserves approximately all pairwise distances. Conversely, T-SNE preserves the topology of the neighbourhood. Only the results for the transformer-based method are illustrated in the main text (see Appendix J for other configurations).

Globally the HDSRS appears well structured (see Figure 6.9) with clusters of similar land-cover nature: croplands are mainly concentrated in the bottom part of the figure, forest on the right side, unvegetated areas in the middle and urban areas on the left side. However, many local inconsistencies can be detected, e.g. MCD12Q1 *Urban and Built-up Lands* is surrounded by water areas. Additionally a slight semantic shift between the
various maps is observed. For instance, MOS and the two OCSGE are only represented on the left side, MCD12Q1 on the right. Some clusters are also observed, such as the OSO crops, far from the rest of arable crops and next to *fruit tree and berry plantation* of CLC. This shift between nomenclatures directly stems from nomenclature-specific ways to describe comparable land-cover. For instance, the CLC nomenclature details many inclusion and exclusion criteria, while those notions do not appear in CGLS.

<figure><img src="image_68.png" /><figcaption>Figure 6.10: T-SNE representation of the HDSRS obtained with a trained on generic text corpora Transformer.</figcaption></figure>

Figure 6.10 illustrates that despite having coherent local neighbours association from a generic point of view, observed association are not necessarily those expected from a land-cover one. For instance, *Natural grassland* and *Pasture* are nearest neighbors in the
HDSRS while most semantic based standardisation system would have consider them far
apart as one describes a natural area and the other croplands.

The same observation can be drawn for the Word2Vec encodings provided (Appendix J).
The BoW encodings distinguishes itself from the others by a pronounced outlier like visual
aspect underlying that a few classes are encoded very differently than the others. The
BoW on average better Inertia and CH thus hides that a few rare classes are encoded
poorly. We link this behaviour to the use of word proportion instead of semantic meaning:
some classes exhibits unique words resulting in a specific encoding.

6.3.2 SRS optimisation through dimension reduction

6.3.2.1 Motivation

The initial results of Step 1 are already satisfactory albeit neighborhood conservation
is not guaranteed. The main reason is that the encoding task is not land-cover specific.
It could be interesting to only extract the relevant features from the 384 dimensions of
the SRS. In particular since such a high dimension leads to technical constraints (32GB
memory GPU). We propose to investigate how to reduce the number of dimensions while
keeping only land-cover relevant elements.

Numerous techniques have been proposed so far [316]: feature selection (some dimensions are removed), matrix projection (linear combination of dimensions), manifold learning (non-linear combination), or auto-encoders (the data is reprojected in fewer dimensions with the constrain of ensuring that the original space can be retrieved). Most are self-supervised exhibiting principally three different objective functions: preserving the distance observed between elements in the HDSRS in the LDSRS (e.g. Isomap [300], Locally Linear Embedding [259]), the nearest neighbours (Neighborhood Preserving Embedding [116]), the information (auto-encoder reconstructs the original HDSRS from the LDSRS). All those objectives try to keep the information of the HDSRS while we point out that many of them are irrelevant from a land-cover point of view. Therefore the LDSRS obtained by those techniques is unlikely to exhibits better metrics than the HDSRS it stems from.

Instead, we propose to investigate supervised dimension reduction techniques using the
LCDD dataset to keep only relevant information. Unlike self-supervised techniques which
target that closeby elements in the HDSRS remain closeby in the LDSRS, we directly target
that closeby elements in LDSRS are closeby in terms of definition (BOM). Additionally,
the proposed method must ensure that no shift is observed between nomenclatures to
ensure good zero-shot properties. As the dimensionality reduction techniques should be
usable under the zero-shot translation setup, the dimension reduction model should be
trained inductively. Consequently, transductive solutions as T-SNE will not be discussed.
### 6.3.2.2 Method

**Overall idea** Instead of preserving HDSRS neighbours like self supervised methods, we propose supervising the dimensional reduction by enforcing the preservation of neighbours in terms of class definition. Thus we aim to create an Absolute neighbor-preserving space. This constraint is implemented by training a dimension reduction model with the same information on the neighbourhood (BOM) required to compute the CN metric. For instance, the dimension reduction model is trained to represent closely in the LDSRS OSO *Broad-leaved* and MOS *Forest* as we labelled them as a neighbour (BOM=1). We detail the chosen dimension reduction model and how to apply neighbourhood constraints on the LDSRS below. Figure 6.11 illustrates the overall framework.

**Dimension reduction model** The model design is based on two principal observations. First, the model is only trained using a small dataset (less than 169 class definitions, as one nomenclature is kept for testing) on a high dimensional space (at least 300). Therefore, the model should exhibit a small number of learnable parameters to limit overfitting. Secondly, the model should be able to apply a non-linear transformation in order to obtain better neighbour preservation in the LDSRS than those observed in the HDSRS. We rely on a small backbone based on a simple 1-hidden layer perceptron. This MLP takes for input the 384 dimensions and reduces them to x-dimension with x < 100. The best x value is experimentally determined and presented in the next section.

We adopt an adversarial training setup, in which a second MLP is tasked to predict the land-cover nomenclature from its encoding in the LDSRS. This setup drastically reduces the risk of nomenclature shift in the resulting LDSRS. The two MLP include dropout layers with high dropout rates (0.3) to regularize the loss and limit overfitting.

<figure><img src="image_69.png" /><figcaption>Figure 6.11: Overall dimension reduction architecture framework in the case a source class $s_i$ expected to be closer from $t_j$ than $t_k$. During optimisation the transformer parameters are fixed (only the two MLP are trained)</figcaption></figure>

**Loss function** The model should learn the notion of neighbour in a binary fashion. Let $s_i''$ denote the representation of source class i in the LDSRS. The loss uses three elements: the source class representation $s_i''$, a neighbour target $t_j''$, and a non-neighbour one $t_k''$.
We aim to obtain $M_2(s_i'', t_j'') < M_2(s_i'', t_k'')$. Instead of constraining the distance value, this constraint focuses on pairwise relative value, which is less restrictive. This weakly constrained optimisation enables obtaining low dimension SRS pretty easily. We implement this distance constraint using the Triplet Loss proposed by [270] given in Equation 6.14. $\epsilon$ represents a margin value arbitrarily fixed. We underline that in its original form, the $M_2$ metric used is the Euclidean distance, but any other distance metric could also be used.

$$ \text{TripletLoss}_{M_2}(s_i'', t_j'', t_k'') = \max(0, M_2(s_i'', t_j'') - M_2(s_i'', t_k'') + \epsilon). \quad (6.14) $$

Optimising both Euclidean and Cosine distance is possible without deteriorating the quality of the results, we combine two triplet loss functions to enable using both metrics in Equation 6.15:

$$ \text{TripletLoss}(s_i'', t_j'', t_k'') = \text{TripletLoss}_{\text{Euclidean}}(s_i'', t_j'', t_k'') + \text{TripletLoss}_{\text{Cosine}}(s_i'', t_j'', t_k'') \quad (6.15) $$

The arbitrary margin value is set to 1 for Euclidean and 0.5 for cosine. The results are insensible to the margin for Euclidean TripletLoss but must be between 0 and 1 for the cosine as a cosine between 0 and one denotes positively correlated variables.

The adversarial training ensures that the classes of different nomenclatures are mapped into the same part of the space. The dimension reduction MLP is optimised to make the Adversarial MLP predicts that the source and target encodings belong to the same nomenclature (*Adversarial*<sub>*mistake*</sub>). In contrast, the adversarial MLP is optimised to predict the contrary (*Adversarial*<sub>*true*</sub>). Let $A(s_i'', t_j'', t_k'')$ denote the output of the adversarial network. We will consider that the Adversarial network should output 1 when considering two classes belonging to the same nomenclature and 0 otherwise. *Adversarial*<sub>*mistake*</sub> and *Adversarial*<sub>*true*</sub> are given in Equation 6.16 and 6.17.

$$ \text{Adversarial}_{\text{mistake}}(s_i'', t_j'') = L_{CE}(A(s_i'', t_j''), 1). \quad (6.16) $$

$$ \text{Adversarial}_{\text{true}}(s_i'', t_j'', t_k'') = L_{CE}(A(s_i'', t_j''), 0) + L_{CE}(A(s_i'', t_k''), 1). \quad (6.17) $$

To sum up, the dimension reduction MLP is optimised using the full loss (Equation 6.18), while the adversarial network is only optimised using the *Adversarial*<sub>*true*</sub> loss.

$$ \text{DimRedLoss}(s_i'', t_j'', t_k'') = \text{TripletLoss}(s_i'', t_j'', t_k'') + \text{Adversarial}_{\text{mistake}}(s_i'', t_j''). \quad (6.18) $$

**Training procedure** The model is trained iteratively using nine of the ten nomenclatures and we evaluate the quality metric on the last nomenclature. The training is carried out at each iteration on approximately 150 definitions embedded into the 384 dimensions. Therefore the model is inherently overfitting despite its small size and the dropout regularisation.
**Comparison with other dimension reduction techniques** We compare our method to the two most commonly used unsupervised dimension techniques: (i) Principal component analysis (PCA) which linearly combines the HDSRS dimensions in a few new uncorrelated dimensions and (ii) Isomap, a non-linear combination of dimension that aims to preserve local neighbourhoods based on geodesic distance measurement.

We also compare to a supervised dimension reduction technique called Latent Dirichlet allocation (LDA). LDA projects the input data into a linear subspace made of the directions maximising the separation between labeled groups. As we do not have access to groups but to neighbours, we define a group as a set of classes of different nomenclatures that are all pairwise neighbours. For instance, OSO Coniferous, MOS forest and CGLS Open Forest are all pairwise neighbours, which can be considered as one group. In order to optimise the results, we reduce the number of classes belonging to multiple groups as much as possible. In particular, we will consider that two classes of the same nomenclature can belong to the same group provided they are neighbours of the same classes in the other nomenclature e.g. OSO *Broad-leaved* and *Coniferous* belong to the same group.

### 6.3.2.3 Results

We first evaluate the number of dimensions required to achieve the best metrics for each of the methods (Figure 6.12). PCA quickly reaches a plateau with few dimensions (around 10), with metrics very close to those obtained for the HDSRS space. The ISOMAP method performs significantly worse than the other methods. PCA performs well but does not provide a LDSRS with better characteristics than the HDSRS.

LDA method achieves better metrics than those of the HDSRS making it an adequate solution. The method is very efficient in small dimension space (10 to 20) and worsens when more dimensions are used, especially Inertia and CH metrics. Under 40 dimensions, the CH metric is better than the MLP one, indicating that the classes of different nomenclatures are slightly more homogeneously distributed.

The MLP dimension reduction consistently outperforms other solutions on all other metrics. Interestingly this statement holds even when comparing the 5-dimensional space obtained by MLP with less reduced space obtained by other techniques. Even though the best results with the MLP method are observed for 60 dimensions, the difference with the 5-dimensional space is minimal ± 2% for all metrics. In particular, we observe an improvement of 10% in CN, and NP compared to the metrics computed directly in the HDSRS space.

We underline that the average 85% CN is significantly lower from the metrics observed on the training nomenclatures (97% CN with five dimensions, 99% with 100). This is easily understandable as the tiny size of our dataset inherently leads to overfitting. Thus, results could be significantly higher with a more extensive dataset.
<figure><img src="image_70.png" /><figcaption>Figure 6.12: Dimension reduction technique metrics under zero-shot constraint depending on the number of output dimensions. For Inertia and CH, lower values denote the best metrics. For clarity metrics from source and target point of view are averaged (e.g. $CN = (CN_{source} + CN_{target})/2$)</figcaption></figure>
## 6.4 Applications to land-cover translation

We highlighted two applications of SRS: zero-shot source and zero-shot target translations. At the time of writing, zero-shot target is still under experiments and will be included later in Appendix. This Section focuses solely on the zero-shot source, i.e. encoding the input land-cover maps into the SRS and training a network to translate them into various targets. At inference time, unseen source maps are encoded and provided to the network to translate them into one of the target map seen during training. It avoids training new translation models when an unseen source map is provided. Additionally, as the unseen map is never used for training, it alleviates the constraint of translating only source maps with partial spatial overlap. The following section first presents the architecture and training procedure used for our zero-shot source configuration. Results and comparison with other methods are discussed in the second part.

### 6.4.1 Using SRS for zero-shot source translation

#### 6.4.1.1 Architecture

The overall architecture, termed **OneEncoder**(Figure 6.13), encode each map into a 15-dimensional SRS. For instance, all pixels of OSO forest are replaced by the semantic encoding of forest obtained from the Transformer+MLP. Secondly, All maps are resampled using a bilinear resampling algorithm to the maximum resolution (10x10m). We underline that using a continuous resampling algorithm instead of a discrete one (nearest neighbour) is directly made possible by semantic encoding. Lastly, the maps are fed to a single U-Net encoder and then decoded by map-specific decoders. The use of a single-encoder instead of multiple ones like for the **MLCT-Net** (Section 5.1) is made possible by the use of semantically encoded inputs instead of one-hot encodings.

<figure><img src="image_71.png" /><figcaption>Figure 6.13: OneEncoder architecture using a 15 dimensional semantic representation space (in orange). A unique encoder is used for all maps while decoders are target specific. Only 2 maps are represented for readability but the architecture can take any number of maps. White arrows denote the self-reconstruction goal.</figcaption></figure>
### 6.4.1.2 Losses

Similarly to the MLCT-Net, we choose to perform q dual objective training: translation and self-reconstruction. Self-reconstruction is important as it enforces the encoder to avoid losing classes that have no direct equivalent in the other training nomenclatures (such as CLC Wetlands). The total loss is obtained by summing the translation and self reconstruction loss computed using Cross-Entropy. Let $S$ be the source map and $T$ be one of the target maps. Let $E(S)$ denote the encoded version of $S$ and $D_T(E(S))$ denote this encoded-decoded into the $T$ nomenclature, the loss is given in Equation 6.19).

$$L(S, T) = L_{\text{tran}} + L_{\text{self-rec}} = L_{\text{CE}}(D_{T}(E(S)), T) + L_{\text{CE}}(D_{S}(E(S)), S). \quad (6.19)$$

### 6.4.1.3 Training procedure

In an ideal setup, the SRS dimension reduction MLP and the network would be trained with hundreds of different maps, enabling the encoder to be trained with thousands of different encoding values. However, the dimension reduction MLP is trained only on 9 nomenclatures (all the LCDD nomenclatures except the one used at inference.) Similarly, the network is only trained on 5 out of the 6 land-cover maps of the MLULC dataset. Consequently, the network is likely to quickly overfit the training data, restraining the ability to analyse unseen during training semantic encodings.

A noise-based data augmentation technique is used to artificially compensate for the small number of maps. Slight jitter is added around each embedding of the HDSRS to regularize the learning of the MLP. We assume that for a given source nomenclature, all the elements closer to class $s_i$ than all the other classes of the source nomenclature belong to this class. Instead of encoding a map with $c$ classes with $c$ distinct semantic encodings, the $c$ classes can be encoded with an infinity of encodings that respect the previous distance constraint. In practice, each source patch is semantically encoded with this random distance-constrained noise using a single noisy-encoded value for each pixels of a given class. Conversely, the same class is encoded differently in two different patches.

This noise is strong from a translation point of view, as it might change the closer class in the target nomenclature. This leads to insert a additional noise constraint. Since the SRS is built using a triplet loss, all elements below half the margin distance from the source encoding are closer to one of the original neighbours than to the non-neighbour elements. By combining with the previous noise definition, we constraint the noise to both ensure that the noisy-encoding is closer to the original encoding than any other source class encoding and at a maximum distance from it of half the margin.

We will denote **OneEncoder** the OneEncoder method used at inference on the same nomenclature used for training (not zero-shot like in previous chapters). We denote **ZeroShot** the OneEncoder used at inference on a different nomenclature than those used for training (zero-shot source).
### 6.4.1.4 Comparisons

**SemEnc** computes the translation directly from the semantically encoded source map using the Transformer followed by the MLP. Each source class is translated into the closest target class in the SRS without considering the spatial context. It is analogous to the **HardSem** method (Section 4.1) as the translation is performed at the nomenclature level (all pixel of source class *i* are translated the same way and based on semantic assumptions, here on proximity in the SRS space).

**HardSem** is the only baseline that does not require the computation of a model based on pairs of spatially overlapping patches and is thus used as the current reference in terms of zero-shot translation.

**HardStat** aims to associate each source class to each most frequently co-occurring targets. This method is not zero-shot, as the co-occurrence is directly determined from pairs of overlapping maps. It provides an interesting comparison to evaluate if the zero-shot source context-wise model only uses the knowledge acquired during training on target class probability of occurrence. For instance, a model trained to perform CLC/OSO translations will learn that amongst the 11 agricultural classes of the OSO map *Cereals* and *Pastures* are the most frequent. At inference, when translating MOS to OSO, the model could preferably translate all MOS *Crops* pixels to *Cereals* based on this prior knowledge.

The **MLCT-Net** provides an intuition on the upper bound of obtainable results as trained on all nomenclatures. At inference, with unseen source map, **ZeroShot** results should be at best equivalent but more probably lower than the **MLCT-Net**. Conversely, when processing seen source nomenclatures, **OneEncoder** and **MLCT-Net** results should be equivalent. Obtaining on-par results on seen during training maps would indicate that using a single encoder is sufficient to encode any map despite a resolution shift.

## 6.4.2 Results

### 6.4.2.1 Qualitative analysis

Figure 6.14 and Figure 6.15 present the results of the zero-shot translation. **SemEnc** results illustrate well the quality and limitation of the SRS space as they are obtained by associating each source class to the closest target class. First, **SemEnc** translations are often different from those of **HardSem**. Most of the time, differences are observed for source classes that establish multiple semantically equivalent translations. For instance, CGLS *Cropland* are respectively translated into OSO *Cereals* or *Tubers* by **HardSem** and **SemEnc** (second column of Figure 6.14). From a purely semantic point of view, any of those translations is equally valid (as the translation into *Soy* or *Corn*). In practice, one unique translation must be chosen when performing non context-wise translations. We underlined in Section 4.1 that we made the arbitrary choice to choose the most frequently observed class in order to maximize the *HardSem* statistic when multiple equally valid
<figure><img src="image_72.png" /><figcaption>Figure 6.14: Comparison between translation using the baselines and the **ZeroShot** model.</figcaption></figure>
<figure><img src="image_73.png" /><figcaption>Figure 6.15: Comparison between translation using the baselines and the **ZeroShot** model.</figcaption></figure>
semantic translations are possible. Conversely, no such choice is made for the **SemEnc** method: it outputs a semantically consistent class while not necessarily being the most frequent one. Consequently, from a purely quantitative point of view, the **HardSem** method will perform better than the simple **SemEnc** translation.

**ZeroShot** results are highly differs from the other methods. They efficiently retrieve some spatial context information (e.g. third column of Figure 6.14, an OSO patch localized in a mountainous area with *Broad-leaved* and *Natural Grasslands* is translated into OCSGE use). From a purely semantic point of view, *Natural Grasslands* should be translated as *No-use* in the OCSGE use and is translated as such by the **HardSem** and **SemEnc** method. As most OSO *Natural Grasslands* are spatially co-occurring with OCSGE use *Forestry areas*, the **HardStat** method translates them into it. The **MLCT-Net** and **ZeroShot** methods translates them into *Forestry areas* when close to forested areas or when being narrow and linear shaped. They also translate accurately the parts corresponding to grassland in valley areas into *Agriculture*. The comparison between **MLCT-Net** and **ZeroShot** results reveals that the **ZeroShot** framework tends to output less buffered results with a finer geometric aspect. This is particularly visible on linear structures such as the MOS *roads* (last column of Figure 6.14). This context-retrieval ability is observed on numerous other examples such as the distinction between CGLS open and Closed forest (second column of Figure 6.15), or the convincing translation of OSO *Pastures* area between forest patches, near cities and alongside the roads (third column of Figure 6.15).

However, we observe that **ZeroShot** performs poorly when the target is CLC (first and the last column of Figure 6.15). Unlike the other methods that do not apply (**HardSem**, **HardStat**) or underestimate CLC minimum mapping unit (**MLCT-Net**), **ZeroShot** tends to eliminate too much information. We link this behaviour to the fact that learning such coarse MMU rules using a single encoder trained with various resolutions is a difficult task. Conversely, when CLC (first column of Figure 6.14) is translated into a fine resolved map (OSO), we observe a strange linear filamentous-like pattern in the **ZeroShot** results for Vineyards. Once again, we impute this behaviour to the significant resolution gap, making it difficult for a single encoder to perform accurate shape retrieval.

### 6.4.2.2 Quantitative analysis

Table 6.2 compares the results of the **MLCT-Net** (without geographic coordinates encoding) and the **OneEncoder** method. **OneEncoder** results appear almost identical to the **MLCT-Net** method (± 1% OA<sub>a</sub>g and mF1<sub>a</sub>g). When the two networks are trained and tested using the six LULC of the MLULC dataset, replacing the six encoders with a single one is only very slightly deteriorating the quality of the results while dividing the number of parameters by 6 (the lightweight decoders are negligible). This underlines that comparable features can be used to translate all maps despite the resolution gap between the land covers. This observation seems in adequation with traditional image classification convolution neural networks that are often trained using multiple zoom levels
of comparable objects on a single architecture.

<table><caption>Table 6.2: Translation using multiple encoders (MLCT-Net) or a single universal one (OneEncoder)</caption><thead><tr><th rowspan="2">Source<br>Target</th><th rowspan="2"></th><th colspan="5">P</th><th colspan="5">C</th><th colspan="5">O</th><th colspan="5">G1</th><th colspan="5">G2</th><th colspan="5">M</th></tr><tr><th>C</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>O</th><th>G2</th><th>P</th><th>C</th><th>O</th><th>G1</th><th>P</th><th>C</th><th>O</th></tr></thead><tbody><tr><td rowspan="2">OA<sub>ag</sub></td><td>OneEncoder</td><td><b>62</b></td><td>54</td><td>67</td><td>77</td><td><b>76</b></td><td><b>73</b></td><td><b>58</b></td><td>71</td><td>79</td><td>78</td><td><b>77</b></td><td>67</td><td>78</td><td>85</td><td><b>85</b></td><td>67</td><td><b>54</b></td><td><b>56</b></td><td>91</td><td>67</td><td>47</td><td>48</td><td><b>79</b></td><td><b>84</b></td><td>81</td><td>62</td></tr><tr><td>MLCT-Net</td><td><b>62</b></td><td><b>55</b></td><td><b>68</b></td><td><b>78</b></td><td><b>76</b></td><td><b>73</b></td><td><b>58</b></td><td><b>72</b></td><td><b>80</b></td><td><b>79</b></td><td><b>77</b></td><td><b>68</b></td><td><b>79</b></td><td><b>86</b></td><td><b>85</b></td><td><b>68</b></td><td>54</td><td>56</td><td><b>92</b></td><td><b>68</b></td><td>49</td><td>49</td><td><b>79</b></td><td><b>84</b></td><td><b>82</b></td><td><b>63</b></td></tr><tr><td rowspan="2">mF1<sub>ag</sub></td><td>OneEncoder</td><td>25</td><td>26</td><td><b>27</b></td><td>18</td><td>31</td><td>54</td><td><b>36</b></td><td>34</td><td>29</td><td>40</td><td>59</td><td>35</td><td>43</td><td>26</td><td><b>53</b></td><td>46</td><td>23</td><td>27</td><td>43</td><td>39</td><td>19</td><td>21</td><td>38</td><td><b>46</b></td><td>27</td><td><b>21</b></td></tr><tr><td>MLCT-net</td><td><b>26</b></td><td><b>28</b></td><td><b>27</b></td><td><b>19</b></td><td><b>32</b></td><td><b>56</b></td><td><b>36</b></td><td><b>35</b></td><td><b>30</b></td><td><b>41</b></td><td><b>60</b></td><td><b>37</b></td><td><b>44</b></td><td><b>27</b></td><td><b>53</b></td><td><b>48</b></td><td><b>24</b></td><td><b>29</b></td><td><b>44</b></td><td><b>40</b></td><td><b>20</b></td><td><b>22</b></td><td><b>40</b></td><td><b>46</b></td><td><b>29</b></td><td><b>21</b></td></tr></tbody></table>

Table 6.3 compares the results of **ZeroShot** with other models. **ZeroShot** obtains, on average better results than **HardSem** method (+6% OA<sub>ag</sub>, +3% mF1<sub>ag</sub>), demonstrating its ability to use context. It is also obtains better than **HardStat** (+3% OA<sub>ag</sub>, +3% mF1<sub>ag</sub>), demonstrating that this higher ability is not solely based on a higher capacity to translate into the global statically most frequent class but on real spatial context analysis. Interestingly, when studying individual source-to-target translation, we observe that the zero-shot translation from CLC to other nomenclatures achieves significantly worse results than using **HardStat** or **HardSem** baselines. We link this behaviour to the fact that the 44 classes of CLC account for almost a third of all the classes of the ten nomenclatures of the LCDD dataset. Thus, under zero-shot circumstances, the SRS built by training on the nine other nomenclatures have considerable difficulties to accurately represent CLC classes. Circumventing this issue would require significantly increasing the LCDD dataset size. Conversely, other translations, such as the **ZeroShot** from OCSGE use to CGLS, often outperforms the baselines (+12% OA<sub>ag</sub>). **ZeroShot** performs significantly worse than **MLCT-Net** trained on the unseen map, underlying that room for improvement still exists.

<table><caption>Table 6.3: Translation under zero-shot constraint. **MLCT-Net** results are displayed as upper bound performance limit. Bold values denote best performing method among HardSem, HardStat and ZeroShot.</caption><thead><tr><th rowspan="2">Source<br>Target</th><th rowspan="2"></th><th colspan="5">P</th><th colspan="5">C</th><th colspan="5">O</th><th colspan="5">G1</th><th colspan="5">G2</th><th colspan="5">M</th><th colspan="5">Average</th></tr><tr><th>C</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>O</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>G1</th><th>G2</th><th>M</th><th>P</th><th>C</th><th>O</th><th>G2</th><th>P</th><th>C</th><th>O</th><th>G1</th><th>P</th><th>C</th><th>O</th><th></th></tr></thead><tbody><tr><td rowspan="4">OA<sub>ag</sub></td><td>HardSem</td><td>52</td><td>42</td><td>56</td><td>70</td><td><b>75</b></td><td>65</td><td>49</td><td>67</td><td>77</td><td><b>79</b></td><td>57</td><td>54</td><td>69</td><td>76</td><td>80</td><td>53</td><td>39</td><td>34</td><td>87</td><td>54</td><td>37</td><td>31</td><td>75</td><td>77</td><td>72</td><td>59</td><td>61</td></tr><tr><td>HardStat</td><td>54</td><td>44</td><td><b>65</b></td><td>70</td><td><b>75</b></td><td>68</td><td><b>55</b></td><td><b>71</b></td><td><b>78</b></td><td><b>79</b></td><td>61</td><td>54</td><td><b>73</b></td><td>80</td><td><b>81</b></td><td>55</td><td>41</td><td>49</td><td><b>89</b></td><td>54</td><td>37</td><td>41</td><td><b>78</b></td><td>80</td><td>77</td><td><b>62</b></td><td>64</td></tr><tr><td>ZeroShot</td><td><b>57</b></td><td><b>49</b></td><td>62</td><td><b>72</b></td><td>72</td><td><b>69</b></td><td>54</td><td>70</td><td><b>78</b></td><td>77</td><td><b>71</b></td><td><b>60</b></td><td>72</td><td><b>81</b></td><td><b>81</b></td><td><b>65</b></td><td>45</td><td><b>51</b></td><td>83</td><td><b>66</b></td><td>45</td><td><b>47</b></td><td>75</td><td><b>85</b></td><td><b>81</b></td><td><b>62</b></td><td><b>67</b></td></tr><tr><td>MLCT-Net</td><td>62</td><td>55</td><td>68</td><td>78</td><td>76</td><td>73</td><td>58</td><td>72</td><td>80</td><td>79</td><td>77</td><td>68</td><td>79</td><td>86</td><td>85</td><td>68</td><td>54</td><td>56</td><td>92</td><td>68</td><td>49</td><td>49</td><td>79</td><td>84</td><td>82</td><td>63</td><td>71</td></tr><tr><td rowspan="4">mF1<sub>ag</sub></td><td>HardSem</td><td><b>13</b></td><td>17</td><td>22</td><td>15</td><td><b>24</b></td><td>46</td><td><b>32</b></td><td><b>36</b></td><td><b>31</b></td><td><b>42</b></td><td>36</td><td>17</td><td>36</td><td>20</td><td>38</td><td>24</td><td>9</td><td>17</td><td><b>27</b></td><td>19</td><td>8</td><td>8</td><td>29</td><td>35</td><td><b>17</b></td><td><b>19</b></td><td>24</td></tr><tr><td>HardStat</td><td><b>13</b></td><td>18</td><td>19</td><td>16</td><td><b>24</b></td><td><b>47</b></td><td><b>32</b></td><td>33</td><td>30</td><td><b>42</b></td><td>33</td><td>16</td><td>34</td><td>20</td><td>38</td><td>26</td><td>10</td><td>20</td><td><b>27</b></td><td>19</td><td>8</td><td>10</td><td>27</td><td>31</td><td>15</td><td>18</td><td>24</td></tr><tr><td>ZeroShot</td><td>11</td><td><b>23</b></td><td><b>24</b></td><td><b>18</b></td><td><b>24</b></td><td>38</td><td>31</td><td>34</td><td>20</td><td>36</td><td><b>39</b></td><td><b>21</b></td><td><b>38</b></td><td><b>24</b></td><td><b>39</b></td><td><b>43</b></td><td><b>13</b></td><td><b>22</b></td><td>20</td><td><b>35</b></td><td><b>15</b></td><td><b>20</b></td><td><b>31</b></td><td><b>41</b></td><td>14</td><td><b>19</b></td><td><b>27</b></td></tr><tr><td>MLCT-net</td><td>26</td><td>28</td><td>27</td><td>19</td><td>32</td><td>56</td><td>36</td><td>35</td><td>30</td><td>41</td><td>60</td><td>37</td><td>44</td><td>27</td><td>53</td><td>48</td><td>24</td><td>29</td><td>44</td><td>40</td><td>20</td><td>22</td><td>40</td><td>46</td><td>29</td><td>21</td><td>35</td></tr></tbody></table>

## 6.5 Discussion

This section investigated the semantic encoding of land cover classes to increase the potential of learnt land-cover translation models. In particular, we build a semantic representation space encoding the class definitions in the form of a continuous variable ensuring the proximity between semantically close classes and proposed a list of criteria and metrics to evaluate it.
A trained on generic text corpora transformer model is used to represent each class definition in the form of 384 continuous variables. We show that this encoding method analysing the whole definition and not sensitive to words outside the dictionary, represents the land cover classes significantly better than the current methods used in literature (BOW, Word2Vec).

Nonetheless, the obtained encoding is insufficient for map translation as class proximity is imperfectly applied and exhibits too many dimensions to be used in one of the convolutional neural network architectures explored in the previous Chapters. We investigate a supervised dimension reduction method to select the semantically meaningful information from a land cover map translation point of view. We optimise the low-dimensional space based on relative proximity between neighbouring and non-neighbouring classes rather than bare distances. The low-dimension encoding obtained adequately represents classes belonging to nomenclatures never seen during training, demonstrating the viability of the process.

We then propose the application of this low-dimensional semantic representation space for the zero-shot source translation problem. A model is trained to translate several semantically encoded maps and tested on unseen ones. We demonstrate that such a model obtains better results than the baselines. However, the difference with the methods trained on all nomenclatures reveals that improvements are still possible.

The proposed SRS embedding model could be further improved. In particular, it is necessary to significantly increase the size of the LCDD definition dataset on which the dimension reduction method is based. Its limited size prevents correctly encoding specific concepts of classes seldom present in the nomenclatures used, such as wetlands. In addition, it is also necessary to increase the size of the data set used to train the MLULC translation model (here, five nomenclatures are used during training and one for testing). Indeed, a translation model trained on numerous maps will better generalise to unseen during training.

From an application point of view, efficiency on the zero-shot target problem is still to be demonstrated. The translation model is directly trained to represent the source land-cover map in the semantic representation space. Thus, the contextual translation map obtained would no longer be a simple representation in the form of a fixed number of classes but a contextual representation in the semantic space. A pixel of a herbaceous area near water would be placed in the semantic space between the two classes allowing to classify the pixel as herbaceous, water or wetland.

Lastly we point out that SRS could be used for other applications than zero-shot translation.
In particular, we believe that such continuous representation is useful for land-cover fusion
or change-resolution as it enables deriving new classes through simple arithmetic operations
e.g. water + grassland = wetland.
# Conclusion

This last chapter presents our concluding remarks. In particular, we summarise this work's key takeaway and offer some perspectives.

## 7.1 Long story short

**Context** Land-cover products exhibit a fixed nomenclature and resolution restraining their potential adoption and re-use to situations where the nomenclature and resolution of the considered product are adapted. Additionally, as each product exhibits widely different specifications, they are hardly interoperable, i.e. one can not easily replace a land-cover product with another one. Numerous standardisation approaches have been conducted to alleviate those issues, increasing their re-usability. However, none of those frameworks has been universally adopted by the remote-sensing community. Therefore, multiple attempts to harmonise existing nomenclatures have been proposed based on the semantic analysis of class definitions. Even though acknowledging multiple possible translations between a source class and all the classes of the target nomenclature (e.g. *Forest* can be translated either into *Broad-leaved* or *Coniferous*), those methods only proceed to the translation into the semantically closest as they do not define in which context a translation is more relevant than another (e.g. *Coniferous* is the best translation in mountainous areas). Moreover, the resolution translation is addressed independently from the nomenclature translation while those notions are intertwined.

**Solutions** We proposed to replace the actual nomenclature-level land-cover translation (NLLCT) with pixel-level land-cover translation. We investigated context-wise solutions that enable reclassifying and changing the resolution of existing land-cover maps. Specifically, we first focused on which information should be considered for land-cover translation. We proposed a convolution neural network (A-UNet) to account for spatial context and incorporate wide-scale geographical information using learnt geographical coordinate features. Even though those methods significantly increase the number and quality of predictable classes compared to NLLCT, they lacked spatial generalisation ability for maps with a small spatial extent. Hence, inspired by the outstanding results of language
models trained to translate multiple languages, we introduced **MLCT-Net**, a first multi-
land-cover translation framework, significantly improving the spatial generalisation ability.
Since the translation between a source and target nomenclatures might be ill-defined
due to a lack of either semantic information in the source map or a higher spatial target
resolution, we present a versatile solution able to merge context-wisely various highly
resolved image sources with map representation. We observed that image and map fusion
considerably improve translation quality. Lastly, we claim that training models to translate
each land cover as the transformation from an independent representation to another is a
real operational burden and requires retraining the translation model for each map. We
underlined that, theoretically, one could infer the translation of an unseen map based on
knowledge acquired on previous maps. We investigated the potential of semantic represen-
tation spaces and introduced metrics to qualify them. We compared different solutions and
proposed a way to adapt them for land-cover translation. We present an application for
zero-shot source translation, better performing than standard NLLCT methods. Table 7.1
offers a comprehensive view of the principal results evaluated on the six land covers of the
MLULC dataset. For simple one-source to one-target translation, our **A-Unet** improved
the state-of-the-art by 10 points both in OA<sub>ag</sub> and mF1<sub>ag</sub> without taking into account
geographical coordinates and by respectively 11 points and 13 points with them. The
multi-land-cover translation training of **MLCT-net** increases the generalisation ability of
3 points OA<sub>gt</sub> and 5 points in mF1<sub>gt</sub> compared with **A-Unet**. The versatile image fusion
architecture significantly improves the quality of the result when the source is finer resolved
than the target (+8% mF1<sub>ag</sub> compared with **MLCT-NeT**, +5% compared with the image
only approach). However, it is irrelevant when the source map is very coarse compared
to the target, as the pure image classification performs identically (+1% compared to
the image-only approach). Lastly, the zero-shot model offers a compromise between the
constraint of fully supervised training and the results obtained by NLLCT methods. We
believe that the results presented in this manuscript are generalisable to other land-cover
maps as our experiments were conducted at a wide scale with very different land-cover
maps.

**Translation difficulty** Beyond the global results, the high heterogeneity of per translation results (from 0 more than 25% mF1<sub>ag</sub> improvement) illustrates that performance mainly depends on the source and target complementarity. As this manuscript deliberately focuses on challenging translation cases with poorly compatible nomenclature and resolution, we gain a broad understanding of the potential of land-cover translation and its limitations. In particular, we pointed out several factors weakening translation performance, such as classes defined by specific temporal patterns or errors in the source and target maps used to train the model. Even though enabling improvement, the straightforward multi-temporal source translation framework and the divergence-based loss for noise correction still require far more research to allow translation at its full potential. Especially investigating how to consider the errors in source and target simultaneously could tremendously improve the translation. That said, in more straightforward and more
realistic translation setups, in which the source and target characteristics are relatively comparable, translation can quickly produce products of quality comparable to image classification-based ones. For instance, when translating OSO into the ESRI land-cover map (23 classes 10m to 9 classes 10m, both obtained from sentinel imagery), the OA and mF1 go up to 90% and 70% respectively without using additional data while needing less than 12 hours to train. This is interesting for operational purposes as it could be used to increase the update rate of some land-cover maps, such as CLC products released only once each six years.

<table><caption>Table 7.1: Main results on the MLULC dataset. Methods with a * are baseline methods existing prior to our work. X denotes metrics not computable for a given method. ag metrics are computed by comparing the translation with the target while gt metrics are computed on ground truth. Average metrics are computed across all the 26 experimented translations for ag and 15 for gt. DownRes (reciprocally UpRes) metrics corresponds to the average metric value taken only on translations where the source is finer (coarser) than the target. SameExt (reciprocally SpaGe) corresponds to the average metric value taken only on translation where the source and target maps have the same spatial extent (reciprocally when the target has a smaller spatial extent), i.e. OA<sub>gt</sub> SpaGe indicates the spatial generalisation ability of the method. As each SameExt/SpaGe/UpRes/DownRes are average results of different translation. Intercomparison between those values is irrelevant.</caption><thead><tr><th rowspan="2"></th><th colspan="3">OA<sub>ag</sub></th><th colspan="3">mF1<sub>ag</sub></th><th colspan="3">OA<sub>gt</sub></th><th colspan="3">mF1<sub>gt</sub></th><th rowspan="2">Time</th></tr><tr><th>Average</th><th>DownRes</th><th>UpRes</th><th>Average</th><th>DownRes</th><th>UpRes</th><th>Average</th><th>SameExt</th><th>SpaGe</th><th>Average</th><th>SameExt</th><th>SpaGe</th></tr></thead><tbody><tr><td>HardSem*</td><td>61</td><td>58</td><td>63</td><td>25</td><td>23</td><td>26</td><td>72</td><td>58</td><td>81</td><td>38</td><td>33</td><td>40</td><td>&lt;1 hours</td></tr><tr><td>HardStat*</td><td>62</td><td>61</td><td>63</td><td>25</td><td>24</td><td>25</td><td>72</td><td>58</td><td>82</td><td>35</td><td>30</td><td>39</td><td>&lt;3 hours</td></tr><tr><td>SoftLearntFreq*</td><td>X</td><td>66</td><td>X</td><td>X</td><td>27</td><td>X</td><td>X</td><td>X</td><td>X</td><td>X</td><td>X</td><td>X</td><td>&lt;3 hours</td></tr><tr><td>A-UNet</td><td>71</td><td>71</td><td>71</td><td>35</td><td>40</td><td>33</td><td>74</td><td>63</td><td>82</td><td>37</td><td>37</td><td>37</td><td>≈ 12 hours</td></tr><tr><td>A-UNet + coord</td><td>72</td><td>73</td><td>72</td><td>38</td><td>44</td><td>34</td><td>X</td><td>65</td><td>X</td><td>X</td><td>41</td><td>X</td><td>≈ 12 hours</td></tr><tr><td>MLCT-Net</td><td>73</td><td>73</td><td>72</td><td>39</td><td>45</td><td>35</td><td>77</td><td>65</td><td>85</td><td>41</td><td>40</td><td>42</td><td>≈ 24 hours</td></tr><tr><td>S2 only</td><td>77</td><td>75</td><td>79</td><td>47</td><td>48</td><td>46</td><td>77</td><td>67</td><td>84</td><td>40</td><td>43</td><td>39</td><td>≈ 12 hours</td></tr><tr><td>MLCT-Net + S2</td><td>79</td><td>76</td><td>80</td><td>49</td><td>53</td><td>47</td><td>77</td><td>72</td><td>81</td><td>44</td><td>50</td><td>40</td><td>≈ 24 hours</td></tr><tr><td>ZeroShot</td><td>67</td><td>67</td><td>67</td><td>30</td><td>31</td><td>30</td><td>74</td><td>61</td><td>82</td><td>37</td><td>35</td><td>38</td><td>≈ 12 hours</td></tr></tbody></table>

## 7.2 Insight on potential improvement for operational use

The quality of the translated products obtained using NLLCT approaches was generally too low to enable translation in real operational setups. Thus, most works either required to fuse multiple source land-cover translations or assumed knowledge of target segmentation to retrieve products with decent quality. We believe the various solutions proposed in this manuscript could enable new use cases introduced in Figure 7.1. As we needed more time to test every one of them, we outlined below different perspectives and improvements that could be applied to our method to achieve each of them.

**Change Detection and Updating** One has access to a source and a target map covering the same spatial extent with a temporal gap and wants to ensure that they have the same nomenclature and resolution to make them comparable. For instance, one wants
<figure><img src="image_74.png" /><figcaption>Figure 7.1: Main possibilities for land-cover map translation. (A) change detection, (B) updating, (C) validation, (D) harmonisation, (E) completion, (F) spatial simplification, (G) spatial improvement, (H) semantic modification.</figcaption></figure>

to study land-cover change between a 2000 and a 2020 map produced using different nomenclature and resolution. We addressed this case in the example of OSO 2018 to CLC 2012 and obtained accuracy up to 82% at level 2 and 69% at level 3. As the OSO map is produced each year, one could thus create a translated version of CLC each year instead of each six years and reduce the year-long process needed to develop the CLC map. We believe the temporal gap is a reasonable amount of noise (the change between the two dates) compared to the noise already in the source and target maps. This is often the case for vast spatially overlapping maps with reasonable temporal gaps like those experimented on in this manuscript. However, more research should be conducted on other situations, especially on informing the network of the temporal gap-induced noise characteristics. In particular, we believe that an active learning-based solution, such as the one proposed by [179], could be used to learn to differentiate temporal noise from other map errors automatically. This could pave the way to proceed to translation on maps with significant temporal gaps (20 years or more) while increasing noise robustness.

**Validation** One could translate high-quality reference land-cover maps into the exact nomenclature and resolution as a target map one wishes to validate. The comparison between the target and this high-quality translation reference could be used to evaluate the quality of the target product. However, as we pointed out, that translation has highly heterogenous per-class results, so providing per-pixel translation uncertainty with the translated map would be necessary. At the time of writing, raw classifier confidence scores are available. However, we believe in the importance of using approaches such as the one proposed by [108] to calibrate this confidence score into a real probability of errors.

**Harmonisation** One aims to translate multiple source maps into a single target nomenclature and resolution. This is mainly used to compare different maps, but it could also