File size: 14,777 Bytes
88cbcd0 d6cf770 d87c801 d6cf770 4686671 d6cf770 49925a8 9c1de65 09aaa6d 9c1de65 49925a8 9c1de65 49925a8 9c1de65 49925a8 9c1de65 d6cf770 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 09aaa6d 9c1de65 7bf16ce a4fd9ca 58c2f28 64550d3 342859a 58c2f28 a4fd9ca 6dc104b 58c2f28 49925a8 6dc104b 080d6ec 6dc104b 58c2f28 a4fd9ca 0cf331d 49925a8 f214513 1bd51f0 49925a8 56bcf67 f214513 1bd51f0 49925a8 0cf331d 58c706b cd5bb3b 080d6ec cd5bb3b 1bd51f0 49925a8 58c706b 49925a8 58c706b 1bd51f0 49925a8 58c706b 1bd51f0 0cf331d 56bcf67 58c706b 1bd51f0 0cf331d 56bcf67 58c706b 58c2f28 a4fd9ca 58c2f28 080d6ec a4fd9ca 58c2f28 78e38e0 a4fd9ca 0cf331d 1bd51f0 49925a8 58c2f28 0cf331d 3ecce40 a4fd9ca 58c2f28 a4fd9ca 49925a8 080d6ec 49925a8 0cf331d 3ecce40 1bd51f0 3ecce40 a4fd9ca 58c2f28 78e38e0 c9f6e5f 78e38e0 c9f6e5f 78e38e0 c9f6e5f 78e38e0 96776f7 78e38e0 58c2f28 a4fd9ca 080d6ec 49925a8 c9f6e5f e41d9da 64550d3 49925a8 64550d3 49925a8 64550d3 49925a8 e41d9da a4fd9ca 080d6ec 1f57c62 a4fd9ca 3495cd2 080d6ec 49925a8 a4fd9ca 080d6ec 1bce0bc 1f57c62 3495cd2 1f57c62 3495cd2 1bce0bc 3495cd2 49925a8 1f57c62 a4fd9ca f61015f 49925a8 1bce0bc fe474fb 64550d3 49925a8 a4fd9ca cd5bb3b 64550d3 49925a8 a4fd9ca 49925a8 a4fd9ca 58c2f28 a4fd9ca 58c2f28 cd5bb3b 58c2f28 cd5bb3b 58c2f28 cd5bb3b a4fd9ca 58c2f28 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 |
---
license: etalab-2.0
tags:
- segmentation
- pytorch
- aerial imagery
- landcover
- IGN
model-index:
- name: FLAIR-INC_RVBIE_unetresnet34_15cl_norm
results:
- task:
type: semantic-segmentation
dataset:
name: IGNF/FLAIR/
type: earth-observation-dataset
metrics:
- name: mIoU
type: mIoU
value: 58.63
- name: Overall Accuracy
type: OA
value: 76.3711
- name: Fscore
type: Fscore
value: 72.4353
- name: Precision
type: Precision
value: 74.3015
- name: Recall
type: Recall
value: 72.4891
- name: IoU Buildings
type: IoU
value: 82.6313
- name: IoU Pervious surface
type: IoU
value: 53.2351
- name: IoU Impervious surface
type: IoU
value: 74.1742
- name: IoU Bare soil
type: IoU
value: 60.3958
- name: IoU Water
type: IoU
value: 87.5887
- name: IoU Coniferous
type: IoU
value: 46.3504
- name: IoU Deciduous
type: IoU
value: 67.4473
- name: IoU Brushwood
type: IoU
value: 30.2346
- name: IoU Vineyard
type: IoU
value: 82.9251
- name: IoU Herbaceous vegetation
type: IoU
value: 55.0283
- name: IoU Agricultural land
type: IoU
value: 52.0145
- name: IoU Plowed land
type: IoU
value: 40.8387
- name: IoU Swimming pool
type: IoU
value: 48.4433
- name: IoU Greenhouse
type: IoU
value: 39.4447
pipeline_tag: image-segmentation
---
<div style="border:0px; padding:25px; background-color:#F8F5F5; padding-top:10px; padding-bottom:1px;">
<h1>FLAIR model collection</h1>
<p>The FLAIR models are a collection of semantic segmentation models initially developed to classify land cover on very high resolution aerial images (more specifically the French <a href="https://geoservices.ign.fr/bdortho">BD ORTHO®</a> product). The distributed pre-trained models differ in their :</p>
<ul style="list-style-type:disc;">
<li>dataset for training : <a href="https://huggingface.co/datasets/IGNF/FLAIR"><b>FLAIR</b> dataset</a> or the increased version of this dataset <b>FLAIR-INC</b> (x 3.5 patches). Only the FLAIR dataset is open at the moment.</li>
<li>input modalities : <b>RGB</b> (natural colours), <b>RGBI</b> (natural colours + infrared), <b>RGBIE</b> (natural colours + infrared + elevation)</li>
<li>model architecture : <b>resnet34_unet</b> (U-Net with a Resnet-34 encoder), <b>deeplab</b></li>
<li>target class nomenclature : <b>12cl</b> (12 land cover classes) or <b>15cl</b> (15 land cover classes)</li>
</ul>
</div>
<br>
<div style="border:1px solid black; padding:25px; background-color:#FDFFF4 ; padding-top:10px; padding-bottom:1px;">
<h1>FLAIR-INC_RVBIE_resnet34_unet_15cl_norm model</h1>
<p>The general characteristics of this specific model <strong>FLAIR-INC_RVBIE_resnet34_unet_15cl_norm</strong> are :</p>
<ul style="list-style-type:disc;">
<li>Trained with the FLAIR-INC dataset</li>
<li>RGBIE images (true colours + infrared + elevation)</li>
<li>U-Net with a Resnet-34 encoder</li>
<li>15 class nomenclature : [building, pervious surface, impervious surface, bare soil, water, coniferous, deciduous, brushwood, vineyard, herbaceous, agricultural land, plowed land, swimming pool, snow, greenhouse]</li>
</ul>
</div>
## Model Informations
- **Code repository:** https://github.com/IGNF/FLAIR-1
- **Paper:** https://arxiv.org/pdf/2211.12979.pdf
- **Developed by:** IGN
- **Compute infrastructure:**
- software: python, pytorch-lightning
- hardware: HPC/AI resources provided by GENCI-IDRIS
- **License:** : Apache 2.0
---
## Uses
Although the model can be applied to other type of very high spatial earth observation images, it was initially developed to tackle the problem of classifying aerial images acquired on the French Territory.
The product called ([BD ORTHO®](https://geoservices.ign.fr/bdortho)) has its own spatial and radiometric specifications. The model is not intended to be generic to other type of very high spatial resolution images but specific to BD ORTHO images.
Consequently, the model’s prediction would improve if the user images are similar to the original ones.
_**Radiometry of input images**_ :
The BD ORTHO input images are distributed in 8-bit encoding format per channel. When traning the model, input normalization was performed (see section **Trainingg Details**).
It is recommended that the user apply the same type of input normalization while inferring the model.
_**Multi-domain model**_ :
The FLAIR-INC dataset that was used for training is composed of 75 radiometric domains. In the case of aerial images, domain shifts are frequent and are mainly due to : the date of acquisition of the aerial survey (from april to november), the spatial domain (equivalent to a french department administrative division) and downstream radiometric processing.
By construction (sampling 75 domains) the model is robust to these shifts, and can be applied to any images of the ([BD ORTHO® product](https://geoservices.ign.fr/bdortho)).
_**Specification for the Elevation channel**_ :
The fifth dimension of the RGBIE images is the Elevation (height of building and vegetation). This information is encoded in a 8-bit encoding format.
When decoded to [0,255] ints, a difference of 1 should coresponds to 0.2 meters step of elevation difference.
_**Land Cover classes of prediction**_ :
The orginial class nomenclature of the FLAIR Dataset encompasses 19 classes (See the [FLAIR dataset](https://huggingface.co/datasets/IGNF/FLAIR) page for details).
However 3 classes corresponding to uncertain labelisation (Mixed (16), Ligneous (17) and Other (19)) and 1 class with very poor labelling (Clear cut (15)) were desactivated during training.
As a result, the logits produced by the model are of size 19x1, but classes n° 15, 16, 17 and 19 should appear at 0 in the logits and should not be present in the final argmax product.
## Bias, Risks, Limitations and Recommendations
_**Using the model on input images with other spatial resolution**_ :
The FLAIR-INC_RVBIE_resnet34_unet_15cl_norm model was trained with fixed scale conditions. All patches used for training are derived from aerial images with 0.2 meters spatial resolution. Only flip and rotate augmentations were performed during the training process.
No data augmentation method concerning scale change was used during training. The user should pay attention that generalization issues can occur while applying this model to images that have different spatial resolutions.
_**Using the model for other remote sensing sensors**_ :
The FLAIR-INC_RVBIE_resnet34_unet_15cl_norm model was trained with aerial images of the ([BD ORTHO® product](https://geoservices.ign.fr/bdortho)) that encopass very specific radiometric image processing.
Using the model on other type of aerial images or satellite images may imply the use of transfer learning or domain adaptation techniques.
_**Using the model on other spatial areas**_ :
The FLAIR-INC_RVBIE_resnet34_unet_15cl_norm model was trained on patches reprensenting the French Metropolitan territory.
The user should be aware that applying the model to other type of landscapes may imply a drop in model metrics.
---
## How to Get Started with the Model
Visit ([https://github.com/IGNF/FLAIR-1](https://github.com/IGNF/FLAIR-1)) to use the model.
Fine-tuning and prediction tasks are detailed in the README file.
---
## Training Details
### Training Data
218 400 patches of 512 x 512 pixels were used to train the **FLAIR-INC_RVBIE_resnet34_unet_15cl_norm** model.
The train/validation split was performed patchwise to obtain a 80% / 20% distribution between train and validation.
Annotation was performed at the _zone_ level (~100 patches per _zone_). Spatial independancy between patches is guaranted as patches from the same _zone_ were assigned to the same set (TRAIN or VALIDATION).
The following number of patches were used for train and validation :
| TRAIN set | 174 700 patches |
| VALIDATION set | 43 700 patchs |
### Training Procedure
#### Preprocessing
For traning the model, input normalization was performed to center-reduce (**a mean=0** and a **standard deviation = 1**, channel wise) the dataset.
We used the statistics of TRAIN+VALIDATION for input normalization. It is recommended that the user apply the same type of input normalization.
Statistics of the TRAIN+VALIDATION set :
| Modalities | Mean (Train + Validation) |Std (Train + Validation) |
| ----------------------- | ----------- |----------- |
| Red Channel (R) | 105.08 |52.17 |
| Green Channel (G) | 110.87 |45.38 |
| Blue Channel (B) | 101.82 |44.00 |
| Infrared Channel (I) | 106.38 |39.69 |
| Elevation Channel (E) | 53.26 |79.30 |
#### Training Hyperparameters
* Model architecture: Unet (implementation from the [Segmentation Models Pytorch library](https://segmentation-modelspytorch.readthedocs.io/en/latest/docs/api.html#unet))
* Encoder : Resnet-34 pre-trained with ImageNet
* Augmentation :
* VerticalFlip(p=0.5)
* HorizontalFlip(p=0.5)
* RandomRotate90(p=0.5)
* Input normalization (mean=0 | std=1):
* norm_means: [105.08, 110.87, 101.82, 106.38, 53.26]
* norm_stds: [52.17, 45.38, 44, 39.69, 79.3]
* Seed: 2022
* Batch size: 10
* Number of epochs : 200
* Early stopping : patience 30 and val_loss as monitor criterium
* Optimizer : SGD
* Schaeduler : mode = "min", factor = 0.5, patience = 10, cooldown = 4, min_lr = 1e-7
* Learning rate : 0.02
* Class Weights : [1-building: 1.0 , 2-pervious surface: 1.0 , 3-impervious surface: 1.0 , 4-bare soil: 1.0 , 5-water: 1.0 , 6-coniferous: 1.0 , 7-deciduous: 1.0 , 8-brushwood: 1.0 , 9-vineyard: 1.0 , 10-herbaceous vegetation: 1.0 , 11-agricultural land: 1.0 , 12-plowed land: 1.0 , 13-swimming_pool: 1.0 , 14-snow: 1.0 , 15-clear cut: 0.0 , 16-mixed: 0.0 , 17-ligneous: 0.0 , 18-greenhouse: 1.0 , 19-other: 0.0]
#### Speeds, Sizes, Times
The FLAIR-INC_RVBIE_resnet34_unet_15cl_norm model was trained on a HPC/AI resources provided by GENCI-IDRIS (Grant 2022-A0131013803).
16 V100 GPUs were used ( 4 nodes, 4 GPUS per node). With this configuration the approximate learning time is 6 minutes per epoch.
FLAIR-INC_RVBIE_resnet34_unet_15cl_norm was obtained for num_epoch=76 with corresponding val_loss=0.56.
<div style="position: relative;">
<p>TRAIN loss</p>
<img src="train_loss_FLAIR-INC_RGBIE_resnet34_unet_15cl_norm.png" alt="drawing" style="width: 60%;"/>
<p>VALIDATION loss</p>
<img src="val_loss_FLAIR-INC_RGBIE_resnet34_unet_15cl_norm.png" alt="drawing" style="width: 60%;"/>
</div>
## Evaluation
### Testing Data, Factors & Metrics
#### Testing Data
The evaluation was performed on a TEST set of 31 750 patches that are independant from the TRAIN and VALIDATION patches. They represent 15 spatio-temporal domains.
The TEST set corresponds to the reunion of the TEST set of scientific challenges FLAIR#1 and FLAIR#2. See the [FLAIR challenge page](https://ignf.github.io/FLAIR/) for more details.
The choice of a separate TEST set instead of cross validation was made to be coherent with the FLAIR challenges.
However the metrics for the Challenge were calculated on 12 classes and the TEST set acordingly.
As a result the _Snow_ class is absent from the TEST set.
#### Metrics
With the evaluation protocol, the **FLAIR-INC_RVBIE_resnet34_unet_15cl_norm** have been evaluated to **OA= 76.37%** and **mIoU=58.63%**.
The _snow_ class is discarded from the average metrics.
The following table give the class-wise metrics :
| Modalities | IoU (%) | Fscore (%) | Precision (%) | Recall (%) |
| ----------------------- | ----------|---------|---------|---------|
| building | 82.63 | 90.49 | 90.26 | 90.72 |
| pervious surface | 53.24 | 69.48 | 68.97 | 70.00 |
| impervious surface | 74.17 | 85.17 | 86.28 | 84.09 |
| bare soil | 60.40 | 75.31 | 80.49 | 70.75 |
| water | 87.59 | 93.38 | 93.16 | 93.61 |
| coniferous | 46.35 | 63.34 | 63.52 | 63.16 |
| deciduous | 67.45 | 80.56 | 77.44 | 83.94 |
| brushwood | 30.23 | 46.43 | 63.55 | 36.58 |
| vineyard | 82.93 | 90.67 | 91.35 | 89.99 |
| herbaceous vegetation | 55.03 | 70.99 | 70.59 | 71.40 |
| agricultural land | 52.01 | 68.43 | 59.18 | 81.12 |
| plowed land | 40.84 | 57.99 | 68.28 | 50.40 |
| swimming_pool | 48.44 | 65.27 | 81.62 | 54.37 |
| _snow_ | _00.00_ | _00.00_ | _00.00_ | _00.00_ |
| greenhouse | 39.45 | 56.57 | 45.52 | 74.72 |
| **average** | **58.63** | **72.44** | **74.3** | **72.49** |
The following illustration gives the resulting confusion matrix :
* Left : normalised acording to columns, columns sum at 100% and the **precision** is on the diagonal of the matrix
* Right : normalised acording to rows, rows sum at 100% and the **recall** is on the diagonal of the matrix
<div style="position: relative;">
<p>Normalized Confusion Matrix (precision)</p>
<img src="FLAIR-INC_RVBIE_resnet34_unet_15cl_norm_cm-precision.png" alt="drawing" style="width: 70%;"/>
<p>Normalized Confusion Matrix (recall)</p>
<img src="FLAIR-INC_RVBIE_resnet34_unet_15cl_norm_cm-recall.png" alt="drawing" style="width: 70%;"/>
</div>
### Results
Samples of results
---
## Citation
**BibTeX:**
```
@inproceedings{ign-flair,
title={FLAIR: a Country-Scale Land Cover Semantic Segmentation Dataset From Multi-Source Optical Imagery},
author={Anatol Garioud and Nicolas Gonthier and Loic Landrieu and Apolline De Wit and Marion Valette and Marc Poupée and Sébastien Giordano and Boris Wattrelos},
year={2023},
booktitle={Advances in Neural Information Processing Systems (NeurIPS) 2023},
doi={https://doi.org/10.48550/arXiv.2310.13336},
}
```
**APA:**
```
Anatol Garioud, Nicolas Gonthier, Loic Landrieu, Apolline De Wit, Marion Valette, Marc Poupée, Sébastien Giordano and Boris Wattrelos. 2023.
FLAIR: a Country-Scale Land Cover Semantic Segmentation Dataset From Multi-Source Optical Imagery. (2023).
In proceedings of Advances in Neural Information Processing Systems (NeurIPS) 2023.
DOI: https://doi.org/10.48550/arXiv.2310.13336
```
## Contact : ai-challenge@ign.fr
|