SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anatol Garioud<sup>1</sup> Nicolas Gonthier<sup>2</sup> Loic Landrieu<sup>2,3</sup> Apolline De Wit<sup>1</sup> Marion Valette<sup>1</sup> Marc Poupee<sup>4</sup> Sébastien Giordano<sup>1</sup> Boris Wattrelos<sup>1</sup>
$^{1}$ French National Institute of Geographical and Forest Information (IGN)
$^{2}$ Univ Gustave Eiffel, IGN, ENSG, LASTIG
$^{3}$ LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS
$^{4}$ National School of Geographic Sciences (ENSG)
{firstname_lastname}@ign.fr
We introduce the French Land cover from Aerospace ImageRy (FLAIR), an extensive dataset from the French National Institute of Geographical and Forest Information (IGN) that provides a unique and rich resource for large-scale geospatial analysis. FLAIR contains high-resolution aerial imagery with a ground sample distance of  $20\mathrm{cm}$  and over 20 billion individually labeled pixels for precise land-cover classification. The dataset also integrates temporal and spectral data from optical satellite time series. FLAIR thus combines data with varying spatial, spectral, and temporal resolutions across over  $817\mathrm{km}^2$  of acquisitions representing the full landscape diversity of France. This diversity makes FLAIR a valuable resource for the development and evaluation of novel methods for large-scale land-cover semantic segmentation and raises significant challenges in terms of computer vision, data fusion, and geospatial analysis. We also provide powerful uni- and multi-sensor baseline models that can be employed to assess algorithm's performance and for downstream applications. Through its extent and the quality of its annotation, FLAIR aims to spur improvements in monitoring and understanding key anthropogenic development indicators such as urban growth, deforestation, and soil artificialization. Dataset and codes can be accessed at https://ignf.github.io/FLAIR/
1 Context
According to a 2015 report by the Food and Agriculture Organization of the United Nations (FAO) [1], approximately  $75\%$  of the world's soils are in fair, poor, or very poor condition. This degradation poses significant threats to the health and long-term sustainability of ecosystems. Healthy soils provide invaluable ecosystem services, including: (i) providing natural habitats for numerous plant and animal species [2], (ii) acting as the largest carbon sink, surpassing the atmosphere and all combined biomass [3], and (iii) functioning as a rainwater reservoir, supporting food production and storing freshwater [4].
The degradation of soils and biodiversity is largely attributed to land artificialization [1], which causes long-term damage to the biological, hydrological, climatic, and agronomic functions of the soil due to its occupation or use [5, 6]. In order to effectively monitor and manage land artificialization, public authorities have expressed the need for scalable land-cover monitoring tools. With the increasing availability of high-quality Earth Observation (EO) data, the French National Institute
Figure 1: Detail from the FLAIR Dataset. The very high resolution annotation of  $20\mathrm{cm}$  allows us to distinguish the exact extent of individual houses, roads, and trees.
of Geographical and Forest Information (IGN) [7] is exploring the use of artificial intelligence to automatically conduct high-resolution, global-scale land-cover mapping, an essential component for addressing soil degradation at a national level [8]. A central aspect of this initiative is to create and disseminate precise and up-to-date reference datasets for researchers and policy-makers.
We introduce the French Land cover from Aerospace ImageRy (FLAIR) dataset, the largest multi-sensor land-cover dataset with very-high-resolution annotations. FLAIR combines very-high-resolution (VHR, 20cm) images, photogrammetry-derived surface models, and optical Sentinel-2 multi-spectral satellite time series with a nominal revisit time of 5 days at the equator. The diverse spatial, spectral, and temporal resolutions of these acquisitions offer valuable complementary perspectives for land cover analysis. Over 20 billion pixels have been hand-annotated by geospatial experts, using a nomenclature of 19 land-cover classes. The data spans  $817\mathrm{km}^2$  across 50 French sub-regions featuring diverse bioclimatic attributes at various times of the year, thus displaying complex and challenging domain shifts.
FLAIR combines data sources with heterogeneous spatial, temporal, and spectral resolutions and high-precision annotations, and aims to foster the development of new large-scale semantic segmentation methods. Given its scale and the complexity of the domain shifts it exhibits, FLAIR also presents an exciting challenge for the computer vision and machine learning communities.
Numerous land-cover datasets have been introduced to train semantic segmentation methods, see Table 1. Existing datasets usually present a trade-off: they either offer high-resolution annotations but cover a small extent (like Vaihingen [9]), or provide large-extent coverage but with low-resolution annotations (such as BigEarthNet [10] or SEN12MS [11]). In contrast, FLAIR offers both very high-resolution annotations (20 cm) while covering a large portion of the French territory.
FLAIR comprises over 20 billion individually, manually annotated pixels, which is over 1000 times more than SEN12MS and 2 times more than BigEarthNet, which employs semi-automatic annotations. The DeepGlobe and LoveDA datasets, the closest counterparts to our dataset, provide a large coverage of  $1717\mathrm{km}^2$  at  $50~\mathrm{cm}$  and  $536\mathrm{km}^2$  at  $30~\mathrm{cm}$  respectively. However, FLAIR provides over 3 times as many annotated pixels and a higher resolution.
The spatial resolution of the annotation is crucial in land-cover analysis. Insufficient resolution prevents the precise measurements of surfaces and boundaries. Furthermore, small-scale features, such as individual houses, lone trees or roads, may not be captured accurately, limiting the potential applications of the derived segmentation.
Table 1: Land Cover Datasets. Publicly available datasets for semantic segmentation of land cover from remote sensing Earth observation imagery.
Dataset	Annotation	Acquisition
Pixels × 106	Resolution	Classes	Source	Resolution	Extent (km2)	Source
SAT-4/SAT-6 [12]	0.9	28 m	4/6	semi-automatic (NLCD [13])	1 m	13860k	aerial
SEN12MS [11]	14	100 m	17	fully-automatic (MODIS [14])	10 m	3551k	Sentinel-1&-2
Vaihingen [9]	82	8 cm	6	visual interpretation	8 cm	1	aerial
EuroSAT [15]	110	50 m	10	EU Urban Atlas [16]	10 m	11059	Sentinel-2
MultiSenGE [17]	534	10 m	14	visual interpretation	10 m	57433	Sentinel-1&-2
Landcovernet [18]	589	10 m	7	semi-automatic (MODIS [14])	10 m	58982	Sentinel-2
MiniFrance [19]	1510	50 m	14	EU Urban Atlas [16]	50 cm	53000	aerial
DynamicEarthNet [20]	1889	3 m	7	visual interpretation	3 m	16986	Sentinel-1&-2, PlanetFusion aerial, UAV, satellite
OpenEarthMap [21]	4931	25–50 cm	8	visual interpretation	25–50 cm	799
Five-Billion-Pixels [22]	5000	4 m	24	visual interpretation	4 m	50000	Gaofen-2
LoveDA [23]	6000	30 cm	7	visual interpretation	30 cm	536	aerial
DeepGlobe [24]	6867	50 cm	7	visual interpretation	50 cm	1717	Wordview-2/3, GeoEye-1
BigEarthNet [10]	8500	100 m	19	semi-automatic (CLC [25])	10 m	850 k	Sentinel-1&-2
FLAIR	20385	20cm	19	visual interpretation	20 cm/10 m	817	aerial, Sentinel-2
FLAIR combines granular pixel annotation with heterogeneous data sources across a large and diverse spatio-temporal extent.
3.1 Extent & Annotation
Spatio-Temporal Distribution. The FLAIR dataset consists of 77 762 patches represented in Figure 3. Each patch includes a high-resolution aerial image of  $0.2\mathrm{m}$ , a yearly satellite image time series with a spatial resolution of  $10\mathrm{m}$ , and pixel-precise elevation and land cover annotations at  $0.2\mathrm{m}$  resolution. As shown in Figure 5, the acquisitions are taken from 916 unique areas distributed across 50 French spatial domains (départements), covering approximately  $817\mathrm{km}^2$ . Aerial images were captured under favorable weather conditions between April and November from 2018 to 2021. Each satellite time series corresponds to the entire year of acquisition of the matching aerial image.
Annotations. Each pixel has been manually annotated by photo-interpretation of the  $20\mathrm{cm}$  resolution aerial imagery, carried out by a team supervised by geography experts from the IGN. During the annotation process, we initially identified 18 classes. We group certain classes together due to the rarity of certain classes, such as swimming pool, greenhouse, or snow, or potential ambiguity, as seen with ligneous and mixed vegetation. The resulting 12-class nomenclature leads to more statistically robust evaluation metrics. Nonetheless, users can still access and use the extended nomenclature.
Table 2: Land-cover Class Distribution. Semantic nomenclature used by the FLAIR dataset and their proportion among the entire dataset.
Class	%	Class	%	Class	%
(1) building	7.1	(6) coniferous	4.3	(11) agricultural land	12.8
(2) pervious surface	7.3	(7) deciduous	17.3	(12) plowed land	3.5
(3) impervious surface	12.1	(8) brushwood	6.3	(13) other	0.8
(4) bare soil	3.1	(9) vineyard	3.0
(5) water	4.5	(10) herbaceous vegetation	18.2
Figure 2: Land Cover Spectral Dynamics. We represent the temporal progression of reflectance values for the red channel (B2) for different land cover types using Sentinel-2 satellite data. Each curve represents the mean reflectance for a land cover types over a subset of the Ardèche department in 2020, with shaded regions indicating the standard deviation. This plot illustrates the distinct spectral dynamics of different land cover types.
Movable objects like cars or boats are annotated according to their underlying cover. Table 2 outlines the class set and their distribution. Refer to the appendix for more details.
Thanks to the high resolution of the aerial images, anthropic structures like roads and buildings can be identified with a high level of detail. Agricultural and natural lands like forests or herbaceous cover, which make up over  $65\%$  of the dataset, can often be challenging to distinguish from images alone. As shown in Figure 2, multispectral satellite time series prove to be particularly effective in characterizing the temporal evolution of plant phenology [26, 27], a key motivation for incorporating these into the dataset.
Training Splits. The dataset is made up of 50 distinct spatial domains, aligned with the administrative boundaries of the French départements. For our experiments, we designate 32 domains for training, 8 for validation, and reserve 10 as the official test set (refer to Figure 5 or appendix). This arrangement ensures a balanced distribution of semantic classes, radiometric attributes, bioclimatic conditions, and acquisition times across each set. Consequently, every split accurately reflects the landscape diversity inherent to metropolitan France. It is important to mention that the patches come with meta-data permitting alternative splitting schemes, for example focused on domain shifts.
3.2 Acquisitions
FLAIR offers 3 complementary sources of acquisition, each with distinct nature and spatial/spectral/temporal resolutions: aerial images, elevation models, and satellite image time series.
Very High Resolution Aerial Images. The aerial images are taken from IGN's free license BD-Ortho product [28]. All aerial images are  $512 \times 512$  in size with a resolution of  $20\mathrm{cm}$  per pixel, and feature 4 spectral channels: red, blue, green, and near-infrared. Each patch comes with metadata such as the date and time of acquisition, geographical location and altitude of the patch centroid, and specifics about the camera used for acquisition. All images are aligned to a shared cartographic coordinate reference system (EPSG:2154), and radiometric corrections are applied to ensure homogeneity per spatial domain [29]. This means that colors should not be interpreted as physical measurements of channel reflectance.
Elevation. Each aerial image is accompanied by an elevation value produced by the IGN. This information is not an independent measurement but a product derived from a digital elevation model and a digital surface model obtained through photogrammetry on the aerial images, thereby ensuring temporal consistency.
Sentinel-2 Time Series. Each patch is associated with a satellite image time series from the Sentinel-2 constellation [30], as shown in Figure 4. Each image in the sequence is of size  $40 \times 40$  with a  $10\mathrm{m}$  pixel resolution, centered around the aerial image. Each pixel is characterized by 10
Figure 3: Patches from FLAIR. The dataset is comprised of 77 762 patches. Each patch contains (i) a  $512 \times 512$  aerial image at  $0.2\mathrm{m}$  resolution with red, green, blue (RGB) and near-infrared (NIR) values, (ii) a pixel-precise digital surface model providing an elevation for each pixel, (iii) semantic labels for each pixel, and (iv) an optical time series of spatial dimension  $40 \times 40$  and  $10\mathrm{m}$  per pixel, centered on the aerial image.
spectral bands, ranging from the visible to the medium infrared spectrum; additional details can be found in the appendix. The time series span the entire year during which the corresponding aerial image was acquired and contain from 20 to 110 images, depending on the satellite availability and the orbit characteristics. We include acquisitions with cloud cover and provide cloud and snow probability masks obtained with Sen2cor [31] in the metadata, alongside information about the satellite and its orbit.
Only the spatial and temporal extents of the aerial images are annotated. Terrain features that evolve over time, such as changing river banks or tidal patterns, may not be consistent throughout the time series. Nevertheless, the satellite time series provide invaluable spectral and temporal information, as well as a broader spatial context of  $400 \times 400\mathrm{m}$ .
3.3 Specificities
The FLAIR dataset presents specificities often encountered in geospatial analysis but seldom in computer vision: large-scale multi-sensor acquisitions, and complex spatio-temporal domain shifts.
Multi-Sensor. FLAIR combines optical acquisitions with drastically different spatial (0.2m v.s. 10m), spectral (4 v.s. 10 bands), and temporal (single-date v.s. year-long time series) resolutions. The discrepancy makes the task of integrating this diverse yet complementary information into a unified pixel representation a substantial challenge.
Domain-Shifts. The FLAIR dataset spans a large spatio-temporal extent across the entire French metropolitan territory and various seasons over 3 years. This introduces complex prior-shift (for instance, there are more vineyards near Bordeaux than in Normandy), and concept-shift (variations in roof architecture across regions). This last phenomenon can profoundly impact the appearance of natural and agricultural vegetation, and may also affect the sunlight illumination conditions.
Two camera models were used to capture the aerial images Vexcel's Ultracam Eage Mark3 [32] and IGN's CAMv2 [33]. Although generally minor, this can cause slight differences in resolution and
Figure 4: Satellite Image Time Series. We represent a year-long satellite time series (top), and its associated mono-temporal aerial acquisition and corresponding annotations (bottom).
spectral sensitivity. Additionally, all aerial acquisitions undergo radiometric correction to mitigate disparities caused by sunlight and contrast. This can lead to dissimilarities between spatial domains regarding the spectral response of identical materials.
Figure 5: Spatio-Temporal Distribution. Spatial units of the FLAIR dataset (left), and spatial and temporal distribution of the train / validation / test split (right).
We propose a generic yet powerful multi-sensor architecture to serve as a baseline to evaluate the semantic segmentation performance of different approaches.
General Architecture. An effective model for our task needs to capture both detailed textures from the aerial images and complex temporal dynamics from the time series. We propose a network architecture named U-T&T: U-net with Textural and Temporal information. As shown in Figure 6, our model consists of two networks: one operating on high-resolution images with four radiometric channels (red, green, blue, infrared) and one elevation channel, and one network operating on time series. Each network follows the state-of-the-art approach for their respective data-source.
- U-Net (spatial/texture branch): We use a U-Net [34] with a ResNet34 backbone model [35] pre-trained on the ImageNet dataset [36]. We add two channels on the first layers to accommodate near-infrared and elevation pixel values. The weights of these two channels are initialized randomly [37]. This U-Net branch comprises approximately 24.4 million parameters.
- U-TAE (spatio-temporal branch): We employ a U-Net with temporal attention (U-TAE) to process the Sentinel-2 imagery [38]. This model is specifically designed to extract multi-scale
Figure 6: The U-T&T model. Our architecture comprises two modules: (i) a U-TAE network extracts spatiotemporal descriptors from the Sentinel-2 time series, and (ii) a U-Net network processes the aerial and elevation images. We merge both feature maps with a branch fusion module. Both branches are supervised simultaneously with dedicated losses  $\mathcal{L}_{\mathrm{aerial}}$  and  $\mathcal{L}_{\mathrm{sat}}$ .
spatio-temporal feature maps from satellite image time series. The U-TAE branch includes approximately 2.9 million parameters.
Metadata Encoding. Metadata can significantly impact the interpretation of remote sensing acquisitions. To allow the network to model this specificity, we compute the following features: (i) spatial coordinates of the center of the patch (with Fourier features), (ii) altitude from sea level, (iii) year of acquisition (one-hot-encoded), and (iv) camera types (one-hot-encoded). These features are then processed with a Multi-Layer Perceptron (MLP) and concatenated channelwise to the coarsest (innermost) feature map of the encoder of the U-Net branch. For more details, refer to the appendix.
Fusion Module. We propose to fuse the multi-scale feature maps from the two branches at different scales. For each level of the U-Net encoder of width  $C$  and extent  $H \times W$ , our proposed fusion module has two sub-parts:
- Cropped: This sub-module captures local spatio-temporal information from the image time series. We first crop the pixel-level feature map from the U-TAE to the extent of the aerial image. Then, we apply a spatial convolution with a width of  $C$  and interpolate the results to size  $H \times W$ . Finally, we add the resulting tensor to the intermediate feature map of the U-Net encoder.
- Collapsed: This sub-module captures a larger spatial context of the patch from the image time series. We first compute the spatial mean for each channel of the output map of the U-TAE (of size  $40 \times 40$ ). Then, we process the resulting vector with a three-layer MLP to map it to a width of  $C$ . We add this vector to each pixel of the intermediate feature map of the U-Net encoder.
Network Supervision. The U-Net branch produces a prediction of size  $13 \times 512 \times 512$ , which can be directly supervised pixelwise with the loss  $\mathcal{L}_{\mathrm{aerial}}$ , chosen as the categorical cross entropy. The output of the U-TAE branch is of size  $13 \times 40 \times 40$ , and its extent is larger than the annotation. To adjust this, we first crop it around the aerial image into a tensor of size  $13 \times 10 \times 10$ , then use bilinear interpolation to map it to the desired  $13 \times 512 \times 512$  dimension. This allows us to define its loss  $\mathcal{L}_{\mathrm{sat}}$  as the categorical cross entropy as well.
All class weights are set to 1 except for the other class, which is set to 0 in both losses. The weight of the class plowed land is set to 0 in the loss  $\mathcal{L}_{\mathrm{sat}}$  as it corresponds to a transient state of land cover
whose duration is much shorter than the yearly span of the time series. We combine both losses into a single loss  $\mathcal{L}_{\mathrm{T&T}}$  defined as their unweighted sum:
$$
\mathcal {L} _ {\mathrm {T} \& \mathrm {T}} = \mathcal {L} _ {\text {a e r i a l}} + \mathcal {L} _ {\text {s a t}}.
$$
Implementation Details. The baseline is implemented with PyTorch Lightning [39]. The code for U-Net branch is taken from the segmentation-models-PyTorch library [40], and the U-TAE network is from its official repository [38]. We use the default U-TAE parameters, except for larger widths for the encoder and decoder
The network is optimized with stochastic gradient descent, a batch size of 10, and a learning rate of 0.001. We set the maximum number of epochs to 100 and use early stopping with a patience of 30 epochs. We employ several augmentation strategies, such as cloud removal, temporal averaging, or geometric augmentation; see the appendix for additional details. Our model is trained on a cluster of 12 NVIDIA Tesla V100 GPUs with 32 GB memory.
5 Benchmark
Metric. We assess the performance of different configurations of our baseline using the mean intersection-over-union (mIoU) on the first 12 classes, excluding the other class. We train each model 5 times from scratch, allowing us to compute the standard deviation of the performance.
Analysis. We report in Table 3 the performance for several variations of our baseline models. A U-TAE model using only the satellite image time series and whose prediction is upsampled to the resolution of the aerial images leads to much lower performance than its high-resolution counterparts. In contrast, the performance of the multi-sensor U-T&T model is comparable to the simple U-net operating on high-resolution images: 54.7 v.s. 54.9. However, when using appropriate augmentation strategies for both, the performance gap widens: 54.9 to 56.9. Specifically, we evaluated the following strategies:
- FILT. We remove satellite images with a snow or cloud cover of over  $60\%$  according to the meta-data (with a probability threshold of 0.5). This led to a gain of  $+0.3$  mIoU point on average.
- AVG M. We compute the monthly average of the cloudless satellite acquisitions, ensuring that the time series have at most 12 elements. While the gain is modest (+0.1 point), this approach decreased the memory usage and the length discrepancy between locations.
- MTD. Our meta-data encoding strategy did not show any impact on either approach.
- AUG. We perform geometric augmentations on the aerial images: flip, resize, and random affine transform. This resulted in a  $+0.6$  point increase in performance.
Table 4 provides per-class IoU scores for the two best runs of U-Net and U-T&T. We report better results for the U-T&T model for all 12 classes. In particular, we observe significant improvements for classes that particularly benefit from temporal information and broader spatial context, such as bare soil, coniferous, and vineyard classes. Conversely, the improvement is smaller for the water and plowed land classes. These classes can change significantly across the year due to tides, shifting river beds, or harvesting events. Consequently, year-long observation may not bring useful information. See Figure 7 for qualitative illustrations. More examples are provided in the appendices.
Decoder Architecture. We evaluated the performance of two other decoder networks in the aerial image branch: FPN [41] and DeepLabV3 [42], while we keep the ResNet34 encoder backbone. We report in Table 5 the results with and without our proposed enhancement strategies. The largest network, DeepLabV3, slightly outperforms the other architecture by  $0.4\mathrm{mIoU}$  points.