SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A.6.2 Extra results
We present in Table 11 the performance of a U-TAE network using only satellite image time series upsampled spatially to the resolution of the aerial images. The lower resolution of this sensor leads to significantly worse results.
Table 11: Quantitative Evaluation. Performance of the U-Net and multi-sensor U-T&T architectures on the test set. Results are averages of 5 runs of each configuration. PARA.: number of parameters of the network, in million; EP.: best validation loss epoch.
INPUT	FILT.	AVG M.	MTD	AUG	PARA.	EP.	mIoU
U-Net	aerial	-	-	X	X	24.4	62	54.7±0.1
+MTD	aerial	-	-	✓	X	24.4	59	54.7±0.2
+MTD +AUG	aerial	-	-	✓	✓	24.4	52	55.2±0.1
U-TAE	sat	X	X	-	-	2.3	16	36.1±0.3
+FILT	sat	✓	X	-	-	2.3	14	35.9±0.9
+FILT +AVG M	sat	✓	✓	-	-	2.3	18	36.9±0.2
U-T&T	aerial+sat	X	X	X	X	27.3	9	54.9±0.7
+FILT	aerial+sat	✓	X	X	X	27.3	11	55.2±1.4
+AVG M	aerial+sat	X	✓	X	X	27.3	10	55.0±0.7
+MTD	aerial+sat	X	X	✓	X	27.3	7	54.9±0.6
+AUG	aerial+sat	X	X	X	✓	27.3	22	55.5±1.5
+FILT +AVG M +MTD +AUG	aerial+sat	✓	✓	✓	✓	27.3	21	56.9±1.1
Figure 12 illustrates the confusion matrix of the best U-T&T model. This confusion matrix is derived by combining all individual confusion matrices per patch and is normalized by rows. The analysis of the confusion matrix shows that the best U-T&T model achieves accurate predictions with minimal confusion in the majority of classes. However, when it comes to natural areas such as bare soil and brushwood, although there is improvement due to the use of Sentinel-2 time series data, a certain level of uncertainty remains. These classes exhibit some confusion with semantically similar classes, indicating the challenge of accurately distinguishing them.
More qualitative examples can be found in Figure 13.
Figure 12: U-T&T best model confusion matrix of the test dataset. The matrix is normalized by rows.
Figure 13: Illustration of patch-wise results. Random results on the FLAIR dataset for the multi-sensor approach U-T&T than the standard U-Net model.
B Datasheet for FLAIR dataset
Q1 For what purpose was the dataset created? Was there a specific task in mind? Was there a particular gap that needed to be filled? Please provide a description.
- The FLAIR dataset is created to train and evaluate models that can predict very-high-resolution land cover maps from diverse data sources with heterogeneous spatial, temporal, and spectral resolutions. The main gap we are addressing is the lack of large-scale data with high-definition annotations.
Q2 Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
- This dataset is presented by the French National Institute of Geographical and Forest Information (IGN), a French public state administrative establishment aiming to produce and maintain geographical information for France. The IGN has the mission to document and measure land-cover on French territory and provides referential geographical datasets, including very-high-resolution aerial images and topographic maps. IGN produces reference data and carries out innovation, research and teaching activities. As part of its innovation activities, the IGN provides the FLAIR dataset to democratize access to large-scale open powerful machine learning models through the research and development of open-source resources.
Q3 Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
- The funding of the FLAIR dataset is  $100\%$  public. This work was sponsored by the Ministry of Ecological Transition (more specifically the Directorate for Planning, Housing and Nature Direction générale de l'aménagement, du logement et de la nature) and the Fund for the transformation of public action (Fonds pour la transformation de l'action publique) from the Minister of the Civil Service. The IGN is funded by the French Ministry of Ecological Transition and the French Ministry of Agriculture.
Q4 Any other comments?
- [N/A]
Q5 What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?
- We provide aerial image with corresponding land cover segmentation along with Sentinel-2 satellite image time series around each aerial patch. The acquisitions are taken from 916 unique areas distributed across 50 French spatial domains (départements), covering approximately  $817~km^2$ . The test labels will be released at the end of the second challenge hosted on CodaLab. We made our baseline codes openly available on the FLAIR GitHub page (https://github.com/IGNF/FLAIR-2-AI-Challenge).
Q6 How many instances are there in total (of each type, if appropriate)?
- We provide 77,762 triplet aerial image, Sentinel-2 time and land cover segmentation. The FLAIR dataset encompasses 20,384,841,728 annotated pixels at a spatial resolution of  $0.20\mathrm{m}$  from aerial imagery with a 19 classes land cover. For each area, a comprehensive one-year record of Sentinel-2 acquisitions is also provided. A further overview of the statistics may be seen in the following annexes.
Q7 Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
- The FLAIR dataset, derived from a larger dataset obtained by IGN for cartographic production upon the request of the French government, serves as a representative sample encompassing approximately one-third of the available data. While the complete dataset covers 64 spatial domains, the FLAIR dataset focuses on 50 domains by excluding
contiguous domains and intra-domain areas. Nevertheless, the selected 50 domains offer comprehensive representation in terms of land cover classes, acquisition dates, and macro-climates, and encompass the metadata associated with the entire dataset. The expertise of IGN was leveraged to ensure the selection of a dataset that is representative and informative.
Q8 What data does each instance consist of?
- Each instance consists of an aerial image. Each image is  $512 \times 512$  in size with a resolution of  $20\mathrm{cm}$  per pixel, and feature 4 spectral channels: red, blue, green, and near-infrared along with an elevation value as fifth channel. Each patch is associated with a satellite image time series from the Sentinel-2 constellation (Drusch et al., 2012) of size  $40 \times 40$  with a  $10\mathrm{m}$  pixel resolution, centered around the aerial image. Each pixel from the Sentinel-2 sequences is characterized by 10 spectral bands.
Q9 Is there a label or target associated with each instance?
- [Yes] We provide a complete pixel-precise land cover segmentation per image (19 classes).
Q10 Is any information missing from individual instances?
[No]
Q11 Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?
Q12 Are there recommended data splits (e.g., training, development/validation, testing)?
- Yes, we provide data splits for reproducing the results of the baselines. The test split has been explicitly selected to address the complex domain shifts of geospatial data.
Q13 Are there any errors, sources of noise, or redundancies in the dataset?
- As the annotations are made through visual interpretation with quality control, some errors are unavoidable, especially for classes that are visually hard to distinguish. Internal quality control with multiple annotations has been performed to limit such errors. There are no redundancies in the dataset, each image covers a distinct area.
Q14 Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?
- This dataset is self-contained and will be stored and distributed by the IGN, a public institute. The dataset is under the Open Licence 2.0 of Etalab.
Q15 Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)?
- [No]. The building class does not contain information that would not be available in other open-access sources, such as the cadaster. We have specifically avoided high-risk areas such as military installations or nuclear plants.
Q16 Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.
Q17 Does the dataset relate to people?
- The dataset may feature pedestrian or individuals, but the resolution of  $20\mathrm{cm/pixel}$  and the aerial perspective is not sufficient to recognize them uniquely.
Q18 Does the dataset identify any subpopulations (e.g., by age, gender)?
Q19 Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset?
- [No]. The resolution of  $20\mathrm{cm/pixel}$  and the aerial perspective is insufficient to recognize them uniquely.
Q20 Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)?
Q21 Any other comments?
Q22 How was the data associated with each instance acquired?
- The aerial images are sampled from the ORTHO HR® imagery collection. It is a mosaic of all the individual images taken during an aerial survey done by IGN and mapped onto a cartographic coordinate reference system. The individual images are projected to the RGE ALTI® DTM, which provides solely the altitude of the ground.
- The Sentinel-2 time series were downloaded from the Sinergise Sentinel-Hub API as Level-2A products (see annexes for more information).
Q23 What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)?
- The IGN selected several acquisition companies through a call for tender with strict specifications.
Q24 If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
- The sampling strategy involved class frequency, acquisition dates distribution, radiometric histogram analysis and geographical location spread. The final sampling based on these comprehensive variables was made manually by experts at the IGN.
Q25 Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
- IGN contracted geography experts from the private sector selected through a public call for tender to annotate the dataset. The quality control of the dataset was carried out by geography experts affiliated with IGN. The creation of the dataset was facilitated by researchers and developers employed by IGN under their work contracts.
Q26 Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)?
- The collection of aerial imagery spanned from 2018 to 2021, which coincides with the duration required for an aerial survey to encompass the entirety of the French territory. Annotations were then applied to the aerial images, aligning with the same time frame. Subsequently, the dataset was created in 2022 after the final processing for both the aerial imagery and annotations.
Q27 Were any ethical review processes conducted (e.g., by an institutional review board)?
Q28 Does the dataset relate to people?
Q29 Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?
[ N/A]
Q30 Were the individuals in question notified about the data collection?
Q31 Did the individuals in question consent to the collection and use of their data?
Q32 If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses?
Q33 Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted?
Q34 Any other comments?
B.4 Preprocessing, Cleaning, and/or Labeling
Q35 Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
Q36 Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
Q37 Is the software used to preprocess/clean/label the instances available?
Q38 Any other comments?
Q39 Has the dataset been used for any tasks already?
- The optical images of FLAIR train split were used for two data challenges ran in 2022 and 2023 by IGN.
- Marsocci et al., 2023 used a subset of FLAIR for to evaluate techniques for unsupervised domain adaptation.
Q40 Is there a repository that links to any or all papers or systems that use the dataset?
- [Yes]. We propose below a list of scientific publications and systems that use FLAIR dataset:
- Garioud et al., 2022 provides a technical description of the FLAIR aerial imagery dataset;
- Garioud et al., 2023 provides insight on the multi-sensor fusion of aerial and satellite imagery;
- Marsocci et al., 2023 experiments remote sensing unsupervised domain adaptation using geographical coordinates on a subset of the FLAIR dataset.
Q41 What (other) tasks could the dataset be used for?
- We encourage future researchers to use FLAIR dataset for several tasks. Particularly, we see applications in land cover segmentation and multi-sensor fusion. Due to the breadth of the data, it also offers a unique opportunity for pre-training of models for other geospatial analysis tasks with low resource, such as object detection, super-resolution, or change detection.
Q42 Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?
- This dataset is geographically limited to metropolitan France. Although France's territory is quite diverse, featuring oceanic, continental, Mediterranean, and mountainous bioclimatic regions, it does not contain tropical or desert areas.
- The FLAIR dataset's reliance on purely optical data may limit the applicability of the models trained on it to regions with pervasive cloud cover.
Q43 Are there tasks for which the dataset should not be used?
- [No].
Q44 Any other comments?
Q45 Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
- [Yes] the dataset will be open-source.
Q46 How will the dataset be distributed (e.g., tarball on website, API, GitHub)?
- The data will be available through .zip files available on the FLAIR project page hosted on GitHub (https://ignf.github.io/FLAIR/).
Q47 When will the dataset be distributed?
- All data with the exception of the test split is presently accessible by registering for an ongoing challenge hosted on Codalab. The entire dataset, including the test split, will be released under an open-source license on the FLAIR project page in early October 2023.
Q48 Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
- [Yes]. The data is governed by the Open Licence 2.0 of Etalab (https://www.etalab.gouv.fr/wp-content/uploads/2018/11/open-licence.pdf).
Q49 Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
Q50 Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?
Q51 Any other comments?
Q52 Who will be supporting/hosting/maintaining the dataset?
- IGN will support hosting of the dataset and metadata.
Q53 How can the owner/curator/manager of the dataset be contacted (e.g., email address)?
ai-challenge@ign.fr
Q54 Is there an erratum?
- [No]. There is no erratum for our initial release. Errata will be documented as future releases on the dataset website.
Q55 Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?
Additional modalities (e.g., supplementary satellite, aerial, UAV-based imagery) may be added to the FLAIR dataset.
Q56 If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)?
Q57 Will older versions of the dataset continue to be supported/hosted/maintained?
- [Yes]. We are dedicated to providing ongoing support for the FLAIR dataset.
Q58 If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?
- Proposed extensions or corrections to the FLAIR dataset may be submitted to the providers for consideration. The IGN will assess the feasibility of incorporating the suggested modifications, considering factors such as data licensing, maintenance requirements, and relevance.
Q59 Any other comments?