SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Quantitative Evaluation. Performance of the U-Net and multi-sensor U-T&T architectures on the test set. Results are averages of 5 runs of each configuration. PARA.: number of parameters of the network, in million; EP.: best validation loss epoch.
INPUT	FILT.	AVG M.	MTD	AUG	PARA.	EP.	mIoU
UTAE	sat	✘	✘	-	-	2.3	16	36.1±0.3
+FILT +AVG M	sat	✓	✓	-	-	2.3	18	36.9±0.2
U-Net	aerial	-	-	✘	✘	24.4	62	54.7±0.1
+MTD +AUG	aerial	-	-	✓	✓	24.4	52	55.2±0.1
U-T&T	aerial+sat	✘	✘	✘	✘	27.3	9	54.9±0.7
+FILT	aerial+sat	✓	✘	✘	✘	27.3	11	55.2±1.4
+AVG M	aerial+sat	✘	✓	✘	✘	27.3	10	55.0±0.7
+MTD	aerial+sat	✘	✘	✓	✘	27.3	7	54.9±0.6
+AUG	aerial+sat	✘	✘	✘	✓	27.3	22	55.5±1.5
+FILT +AVG M +MTD +AUG	aerial+sat	✓	✓	✓	✓	27.3	21	56.9±1.1
Table 4: Per-Class Evaluation. We report the classwise IoU for the best run of the U-Net baseline (aerial imagery), and the U-T&T baseline (aerial and satellite imagery).
model	best	building	perv.	inperv.	bare soil	water	coniferous	deciduous	brushwood	vineyards	herbaceous	agriculture	plowed
U-Net	54.7	80.1	47.3	69.9	30.8	79.9	57.6	70.1	23.9	60.1	46.5	54.5	35.8
U-T&T	58.6	83.5	52.0	74.0	43.7	82.2	64.2	73.7	25.6	64.6	47.17	55.2	37.0
Table 5: Decoder Architecture. We report the performance of different network architectures, with and without our proposed enhancement strategies.
Aerial imagery branch model	enhancements	PARA.	mIoU
U-Net [34]	-	27.3	54.9±0.7
FPN [41]	-	26.1	55.5±1.0
DeepLabV3 [42]	-	28.8	55.8±1.6
U-Net [34]	+FILT +AVG M +MTD +AUG	27.3	56.9±1.1
FPN [41]	+FILT +AVG M +MTD +AUG	26.1	56.2±0.6
DeepLabV3 [42]	+FILT +AVG M +MTD +AUG	28.8	57.3±0.9
Challenges. The FLAIR dataset was used in two CodaLab [43] scientific challenges, both of which received over 1000 submissions, indicating significant interest. The first challenge, from November 2022 to March 2023, involved domain adaptation while the second challenge, from May to September 2023, incorporated Sentinel-2 time series and introduced a new test dataset. These challenges allowed the scientific community to leverage our extensive labeling effort to evaluate, design, and improve large-scale semantic segmentation methods for multi-sensor Earth observation.
Limitations. The FLAIR dataset is limited to metropolitan France. Although France's territory is quite diverse, featuring oceanic, continental, Mediterranean, and mountainous bioclimatic regions, it does not contain tropical or desert area. As a national agency, IGN's focus is limited to France. However, similar efforts by other countries or agencies would increase the diversity of available data and stimulate the design and evaluation of geographically robust methods.
The FLAIR dataset's reliance on purely optical data may limit the applicability of the models trained on it to regions with pervasive cloud cover. Incorporating synthetic aperture radar time series, such as Sentinel-1 time series, may address this limitation and could be considered in a future extension.
Quality Control. As the annotations are made through visual interpretation, some errors are unavoidable, especially for classes that are visually hard to distinguish, such as bare soil and pervious surfaces. We manually annotated around 37k randomly selected polygons, hidden from the annotating
(a) Aerial image
(b) True labels
(c) U-T&T prediction
(d) U-Net prediction
Figure 7: Qualitative illustration. Black ellipses show areas that are significantly better classified by the multi-sensor approach U-T&T than the standard U-Net model.
teams. This covered a total area of  $18.7\mathrm{km}^2$ , equivalent to approximately  $0.75\%$  of the entire dataset, or 468 million pixels. If any batch of annotations did not meet a set accuracy criterion of  $95\%$ , it was rejected and returned for re-annotation. This iterative process fostered productive exchanges between the annotators and the geography experts from IGN, thereby ensuring a high-quality dataset.
Ethics. Releasing a large-scale, high-resolution land-cover dataset openly could raise potential concerns related to privacy, security, and possible misuse. Indeed, detailed information about private properties could be extracted, possibly aiding illegal activities. However, both aerial and satellite images are already publicly available, and we only provide visual interpretations. Furthermore, high-risk areas such as military facilities and nuclear plants are explicitly excluded from the dataset.
FLAIR focuses on the French metropolitan area, which does not well represent countries in arid or tropical climates. This bias could steer scientific efforts towards developing models that perform well in the Western hemisphere while overlooking developing countries. These countries could significantly benefit from automated land-cover tools due to their challenging climates or struggling institutions. IGN is committed to equity and plans future work to focus on more diverse climates, for instance, by planning acquisitions and annotations of French overseas territories.
Potential Social Impact. The primary goal of FLAIR is to stimulate the development of robust and scalable tools for automatic land cover. These tools could be instrumental in monitoring and curbing soil artificialization and its catastrophic environmental impacts. Our dataset can also be used to develop and assess the performance of other key geospatial analysis tasks such as deforestation or sea level monitoring.
By providing a curated and accessible dataset of Earth observations, we also hope to draw the interest of the computer vision community to the challenges of geospatial analysis and contribute to establishing remote sensing data as a standard modality for evaluating machine learning algorithms' performance. The dataset can also facilitate the pre-training of models for other geospatial analysis tasks, such as object detection, super-resolution, or change detection.
7 Licence
FLAIR is under the Open Licence 2.0 of Etalab. This licence has been designed to be compatible with any free licence that at least requires an acknowledgement of authorship, and specifically with the previous version of this licence as well as with the following licences: United Kingdom's "Open Government Licence" (OGL), Creative Commons' "Creative Commons Attribution" (CC-BY) and Open Knowledge Foundation's "Open Data Commons Attribution" (ODC-BY).
The experiments conducted in this study were performed using HPC/AI resources provided by GENCI-IDRIS (Grant 2022-A0131013803). This work was supported by the European Union through the project "Copernicus / FPCUP" as well as by the French Space Agency (CNES) and Connect by CNES. The authors would like to acknowledge the valuable support and resources provided by these organizations.
[33] Jean-Philippe Souchon, Christian Thom, Christophe Meynard, Olivier Martin, and Marc Pierrot-Deseilligny. The IGN CAMv2 system. The Photogrammetric Record, 25(132), 2010.
[34] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, 2015.
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
[36] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 2009.
[37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE international conference on computer vision, 2015.
[38] Vivien Sainte Fare Garnot and Loic Landrieu. Panoptic segmentation of satellite image time series with convolutional temporal attention networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[39] PyTorch Lightning. https://www.pytorchlightning.ai, 2022. [Online; Accessed: 10-October-2022].
[40] Pavel Iakubovskii. Segmentation models Pytorch. https://github.com/qubvel/segmentation_models.pytorch, 2019.
[41] Tsung-Yi Lin, Piotr Dálár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2117-2125, 2017.
[42] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.
[43] Adrien Pavao, Isabelle Guyon, Anne-Catherine Letournel, Xavier Baró, Hugo Escalante, Sergio Escalera, Tyler Thomas, and Zhen Xu. Codalab competitions: An open source platform to organize scientific challenges. Technical report, 2022.
[44] Félix Quinton and Loic Landrieu. Crop rotation modeling for deep learning-based parcel classification from satellite time series. Remote Sensing, 2021.
[45] Sentinel Hub, Sinergise Ltd. https://www.sentinel-hub.com, 2023. [Online; Accessed: 02-May-2023].
[46] SentinelHub. Sentinel2 Cloud Detector. https://github.com/sentinel-hub/sentinel2-cloud-detector, 2023. [Online; accessed 10-August-2023].
[47] Anatol Garioud, Apolline De Wit, Marc Poupee, Marion Valette, Sébastien Giordano, and Boris Wattrelos. Flair #2: textural and temporal information for semantic segmentation from multi-source optical imagery. arXiv, 2023.
[48] Laurent Guigues, Jean Pierre Cocquerez, and Hervé Le Men. Scale-sets image analysis. International Journal of Computer Vision, 2006.
[49] Anatol Garioud, Stéphane Peillet, Eva Bookjans, Sébastien Giordano, and Boris Wattrelos. Flair #1: semantic segmentation and domain adaptation dataset. arXiv, 2022.
A.1 Benefits of the multi-source approach
Sentinel-2 imagery are synergistic approach with VHR aerial images for land cover mapping, as each source has a unique advantage allowing to distinguishing nuanced semantic classes, a critical need in detailed geospatial analysis. Some of the main benefits of integrating Sentinel-2 are:
- Increased spectral resolution: Unlike aerial acquisitions that generally contain only four spectral bands (with a single one in the infrared), Sentinel-2 is furnished with a 10-band multispectral imager. This includes bands in the near-infrared spectrum, which prove essential for discerning vegetation phenology [44].
- Multi-temporal resolution: Sentinel-2 provides a consistent yearly time series. This capability allows our model to trace the temporal progression of each pixel's spectral response, proving invaluable in distinguishing between similar plant species, as depicted in Figure 2. As an illustrative example, while an "agricultural land" and a "herbaceous surface" might appear identical during specific times (exhibiting low herbaceous vegetation), the agricultural land remains barren of vegetation during other parts of the year. VHR aerial acquisitions, in contrast, are limited to single-date images.
- Larger spatial context: The coarser spatial resolution of Sentinel-2 (10 m) compared to aerial images (20 cm) provides an unexpected advantage. By offering a broader context, Sentinel-2 enables our model to harness wider receptive fields. Consequently, each 102x102m aerial patch is linked with a Sentinel-2 image time series spanning a 400x400m area.
- Spectral Consistency: The Sentinel-2 time series benefits from consistent spectral calibration, which aids in countering the radiometric inconsistencies introduced during the BD Ortho's correction process.
A.2 Sentinel-2 Time Series
Table 6 indicates the original bands acquired by the Sentinel-2 satellites and considered in the FLAIR dataset. The images were downloaded from the Sinergise API [45] as Level-2A products (Bottom-Of-the-Atmosphere reflectances) which are atmospherically corrected using the Sen2Cor algorithm [31].
Sentinel-2 sensor acquires images at 10, 20 and  $60\mathrm{m}$  spatial resolutions. The  $60\mathrm{m}$  bands mainly intended for atmospheric corrections are not taken into account and the  $20\mathrm{m}$  bands are resampled during data retrieval to  $10\mathrm{m}$  by the nearest interpolation method.
Table 6: Sentinel-2 spatial and spectral resolutions. Original spatial and spectral resolutions of Sentinel-2 images along with the correspondence between original band number and the distributed data.
Original Band number	FLAIR band number	Central wavelength (nm)	Bandwidth (nm)	Original Spatial resolution (m)	FLAIR Spatial resolution (m)
2	1	490	65	10	10
3	2	560	35	10	10
4	3	665	30	10	10
5	4	705	15	20	10
6	5	740	15	20	10
7	6	783	20	20	10
8	7	842	115	10	10
8a	8	865	20	20	10
11	9	1610	90	20	10
12	10	2190	180	20	10
Table 7 indicates the cloud & snow probability masks provided as separate files alongside the Sentinel-2 acquisitions. It should be noted that cloud detection in satellite images is a complex task because
1 More advanced algorithms [46] could be beneficial.
of the diversity of clouds (thin, scattered clouds). As a result, probability masks can contain errors, notably confusion with surfaces with a high albedo and close to the top of a cloud, as is the case with the roofs of industrial buildings.
Table 7: Provided cloud and snow masks.
Mask	FLAIR band number	Original Spatial resolution (m)	FLAIR Spatial resolution (m)
Snow probability (SNW)	1	20	10
Cloud probability (CLD)	2	20	10
Table 8 provides information about the number of dates included in the filtered Sentinel-2 time series for the train and test datasets. On average, each area is acquired on 55 dates over the course of a year by satellite imagery.
Table 8: Sentinel-2 Time series length. Number of acquisitions (dates) in the Sentinel-2 times series of one year (corresponding to the year of aerial imagery acquisition).
Sentinel-2 time series (1 year)	acquisitions per super-area
min	max	mean
train dataset	20	100	55
test dataset	20	114	55
Note that cloudy dates are not suppressed from the time series. Instead, the masks are provided and can be used to filter the cloudy dates if needed.
The spatial size of Sentinel-2 time series has been empirically determined and set to 40. Nevertheless, we provided in this dataset wider areas than the  $40 \times 40$  used for our baseline. However, there is a limit of 110 pixels for edge patches. The choice of time series spatial size has an impact on the spatial context provided to both the U-TAE and U-Net branches through the collapsed fusion sub-module [47].
A.3 Semantic classes
Overall semantic class number of pixels and frequency of the FLAIR dataset are provided in Table 9. The class distribution in percentages of the train and test sets are presented in Figure 8. The detailed description of the original semantic classes is provided in Table 10.
The ground truth labels are based on photo-interpretation of the aerial imagery at  $20\mathrm{cm}$  and has been manually produced by experts following a call for tenders from the IGN. An initial spatial multi-level image segmentation approach using PYRAM [48] was applied, simplifying the labeling at the small cluster level. This segmentation was modified interactively when deemed appropriate.
Figure 8: Class distribution of the train dataset (left) and test dataset (right).
Table 9: Details about the semantic classes of the main nomenclature of the FLAIR dataset and their corresponding label values, frequency in pixels and percentage among the entire dataset.
Class	Label Value	Pixels	%
building	1	1,453,245,093	7.13
pervious surface	2	1,495,168,513	7.33
impervious surface	3	2,467,133,374	12.1
bare soil	4	629,187,886	3.09
water	5	922,004,548	4.52
coniferous	6	873,397,479	4.28
deciduous	7	3,531,567,944	17.32
brushwood	8	1,284,640,813	6.3
vineyard	9	612,965,642	3.01
herbaceous vegetation	10	3,717,682,095	18.24
agricultural land	11	2,541,274,397	12.47
plowed land	12	703,518,642	3.45
other	>13	153,055,302	0.75
Table 10: Semantic classes of the FLAIR dataset.
Note: as previously stated, semantic classes are assigned on the cluster level. In a given aerial image, only observable objects are labeled, whereby temporal aspects are not taken into consideration.
Anthropized surfaces without vegetation (1, 2, 3, 13 and 18)
Class 1 - building includes not only buildings but also other types of constructions such as towers, agricultural silos, water towers and dams. Greenhouses (class 18) are an exception.
Class 2 - pervious surface defined as man-made bare soils covered with mineral materials (e.g. gravel, loose stones) and considered to be pervious. It includes pervious transport networks (e.g. gravel pathways, railways), quarries, landfills, building sites and coastal ripraps.
Class 3 - impervious surface is defined as man-made bare soils that are impervious due to their building materials (e.g. concrete, asphalt, cobblestones). It includes roadways, parking lots, and certain types of sports fields.
Class 13 - swimming pool is defined as man-made artificial (open-air) swimming pools. It is not included in class 5 (water).
Class 18 - greenhouse although it can be considered as a building, is given a distinct label. Greenhouses are a class of their own and are not part of class 1.
Natural areas without vegetation (4, 5 and 14)
Class 4 - bare soil defined as natural permanently bare soils. These natural soils remain without vegetation throughout the year and generally are covered with sand, pebbles, rocks or stones. Examples of natural bare soils are frequently found in coastal, mountainous and forested areas.
Class 5 - water is defined as areas covered by water, such as sea, rivers, lakes and ponds. An exception are swimming pools (class 13).
Class 14 - snow refers to surfaces covered by snow. It is an extremely rare class as the images are taken in the summertime and only very few regions in France are covered with snow year-round.
Woody natural vegetation surfaces (6, 7, 8, 15, 16 and 17)
Class 6 - coniferous, is defined as trees identifiable as coniferous (pines, firs, cedars, cypress trees, ...) and taller than  $5\mathrm{m}$
Class 7 - deciduous is defined as trees identifiable as deciduous (oaks, beeches, birches, chestnuts, poplars, ...) and taller than  $5\mathrm{m}$ .
Class 8 - brushwood refers to natural woody surfaces with a vegetation less than  $5\mathrm{m}$  high. It includes short and young trees, brushwood, shrublands, mountain moors and abandoned agricultural lands.
Class 15 - clear-cut, is defined as forest areas, in which the trees have been cut down and harvested.
Class 16 - ligneous is an extremely rare class used to describe forest areas with a homogeneous representation of either coniferous or deciduous trees.
Class 17 - mixed is an extremely rare class used to describe forest areas with heterogeneous trees for which the types of trees (coniferous/ deciduous) cannot be determined with sufficient certainty.
Agricultural surfaces (9, 11 and 12)
Class 9 - vineyard despite being an agricultural use of the land, are assigned a class apart, a reason being their rather distinctive land cover characteristics.
Class 11 - agricultural land encompasses various different agricultural classes. For example, besides major crops, it also includes permanent and temporary grasslands with agricultural use. Vineyards (class 9) are not included in this class.
Class 12 - plowed land is defined as agricultural land with no visible vegetation (e.g. recently plowed and freshly harvested land).
Herbaceous surfaces (10)
Class 10 - herbaceous vegetation defines herbaceous surfaces that are not intensively exploited for agriculture purposes. This class includes ornamental lawns (e.g. gardens, public parks), recreational fields (e.g. used for sport), natural herbaceous areas in forested or mountainous areas, non-cultivated grass in agricultural areas or along transportation networks.
A.4 Aerial imagery and spatial domains
Within a spatial domain, all aerial acquisitions are radiometrically corrected to reduce disparities in sunlight and contrast. Nonetheless, this homogenization is not applied equally across all the different spatial domains as can be seen in Figure 9. As opposed to satellite imagery, the pixel intensity in the image channels can therefore not be considered as a physical measure.
Figure 9: Radiometric discrepancies of the aerial images between domains. The 3 channels image displayed is a composite of Near-Infrared, Red and Green spectral information.
A.5 Benchmark architecture
A.5.1 U-Net (spatial/texture branch)
We choose a U-Net architecture [34] with a ResNet34 encoder backbone (pre-trained on the ImageNet dataset [36]) for a total of  $\approx 24.4\mathrm{M}$  parameters and rely on the implementation available in the segmentation-models-pytorch library [40] and trained with the PyTorch lightning [39] framework. The architecture employed is illustrated in Figure 10.
Figure 10: U-Net architecture used for the baseline.  $\mathbf{IMG} =$  input image;  $\mathbf{MTD} =$  input metadata;  $\mathbf{PRED} =$  prediction output. One potential and traditional approach to integrate the metadata would be to add a Multi-layer Perceptron for encoding and add the output to the output of the last layer of the encoder or as an additional band to the IMG input.
Concerning the exploitation of metadata, a simple approach has been tested [49]. The strategies explored have a first step of metadata encoding: positional encoding of spatial and temporal information and one-hot-encoding for camera type and aerial image acquisition year. A shallow MLP with dropout (probability of 0.4) and ReLU activation is then defined to jointly encode the metadata and to provide a specified output size. Subsequently, multiple different integration strategies with the current ResNet34/U-Net segmentation architecture are possible. We have chosen a commonly employed strategy (depicted as 'bottom' in Figure 10) consists in matching the MLP output size to the output size of the last layer of the ResNet34 encoder. The two vectors (encoded metadata and encoded images) can then be added and fed into the first layer of the architecture's decoder. Strategies following similar approaches that add the MLP encoded output at different positions in the architecture's encoder or decoder parts (e.g., after the first input convolution layer, with the last decoder layer, or even added as a sixth channel to the input image) are possible. A positional encoding of size 32 is used specifically for encoding the geographical location information.
The exploitation of metadata deserves to be studied more by the computer vision community, as it could bring real gains by taking advantage of the specificity of remote sensing data.
A.5.2 Fusion module of the U-T&T model
A Fusion Module is employed within the U-T&T baseline model to integrate the feature maps from satellite time-series (with broader spatial extent) into the feature maps from the aerial imagery branch. The details of this module can be seen in Figure 11. Within the Fusion Module, two sub-modules (cropped and collapsed) have different purposes and focus on distinct aspects: the spatio-temporal information and the spatial context. This Fusion Module is applied to match with each feature maps of the U-Net encoder.
Figure 11: Fusion module. This module takes as input the last U-TAE embeddings. It is applied to each stage of the U-Net encoder feature maps. out corresponds to the channel size of the U-Net encoder feature map and  $H$  and  $W$  to the corresponding spatial dimensions.
A.5.3 Data augmentation
By introducing variance in the dataset, image data augmentation helps to prevent overfitting and provides trained models with enhanced generalization capabilities.
For our baseline, only geometric transformations are explored using the Albumentation library. Vertical and horizontal flips, and random rotations of 0, 90, 180 or 270 degrees are tested. A data augmentation probability of 0.5 is used.
A.6 Benchmark results
A.6.1 Official data split of the FLAIR dataset
The following per domain split of the data has been used for the experiments:
TRAIN:	D006, D007, D008, D009, D013, D016, D017, D021, D023, D030, D032, D033, D034, D035, D038, D041, D044, D046, D049, D051, D052, D055, D060, D063, D070, D072, D074, D078, D080, D081, D086, D091
VALIDATION:	D004, D014, D029, D031, D058, D066, D067, D077
TEST:	D015, D022, D026, D036, D061, D064, D068, D069, D071, D084