File size: 14,774 Bytes
5cd9533
060bac3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
eb03c8c
060bac3
 
5cd9533
060bac3
 
 
 
 
 
 
 
6ce03ee
060bac3
 
 
 
 
 
dacd6e7
37c1545
060bac3
 
50bfd21
37c1545
060bac3
 
 
 
 
 
 
 
 
 
 
 
50bfd21
060bac3
 
 
 
 
 
 
 
 
 
dacd6e7
060bac3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
37c1545
060bac3
 
 
37c1545
060bac3
 
 
37c1545
060bac3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
753ee8e
060bac3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
200b7da
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
060bac3
 
 
d0120df
060bac3
 
d0120df
060bac3
 
 
 
5cc8c4e
060bac3
5cc8c4e
060bac3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
eb03c8c
060bac3
 
 
 
eb03c8c
060bac3
eb03c8c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
060bac3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11824c9
060bac3
11824c9
060bac3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
dde7216
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
---
license: etalab-2.0
tags:
- segmentation
- pytorch
- aerial imagery
- landcover
- IGN
model-index:
- name: FLAIR-INC_rgbi_15cl_resnet34-unet
  results:
  - task:
      type: semantic-segmentation
    dataset:
      name: IGNF/FLAIR/
      type: earth-observation-dataset
    metrics:
    - name: mIoU
      type: mIoU
      value: 60.718
    - name: Overall Accuracy
      type: OA
      value: 76.259
    - name: Fscore
      type: Fscore
      value: 74.396
    - name: Precision
      type: Precision
      value: 75.944
    - name: Recall
      type: Recall
      value: 74.033
      
    - name: IoU Buildings
      type: IoU
      value: 78.614
    - name: IoU Pervious surface
      type: IoU
      value: 52.657
    - name: IoU Impervious surface
      type: IoU
      value: 72.566
    - name: IoU Bare soil
      type: IoU
      value: 59.656
    - name: IoU Water
      type: IoU
      value: 87.604
    - name: IoU Coniferous
      type: IoU
      value: 62.255
    - name: IoU Deciduous
      type: IoU
      value: 71.762
    - name: IoU Brushwood
      type: IoU
      value: 31.270
    - name: IoU Vineyard
      type: IoU
      value: 76.454
    - name: IoU Herbaceous vegetation
      type: IoU
      value: 51.643   
    - name: IoU Agricultural land
      type: IoU
      value: 57.639
    - name: IoU Plowed land
      type: IoU
      value: 43.452  
    - name: IoU Swimming pool
      type: IoU
      value: 41.260    
    - name: IoU Greenhouse
      type: IoU
      value: 63.222
      
pipeline_tag: image-segmentation
---


<div style="border:0px; padding:25px; background-color:#F8F5F5; padding-top:10px; padding-bottom:1px;">
  <h1>FLAIR model collection</h1>
  <p>The FLAIR models are a collection of semantic segmentation models initially developed to classify land cover on very high resolution aerial images (more specifically the French <a href="https://geoservices.ign.fr/bdortho">BD ORTHO®</a> product). The distributed pre-trained models differ in their :</p>
  <ul style="list-style-type:disc;">
    <li>dataset for training : <a href="https://huggingface.co/datasets/IGNF/FLAIR"><b>FLAIR</b> dataset</a> or the increased version of this dataset <b>FLAIR-INC</b> (x 3.5 patches). Only the FLAIR dataset is open at the moment.</li>
    <li>input modalities : <b>RGB</b> (natural colours), <b>RGBI</b> (natural colours + infrared), <b>RGBIE</b> (natural colours + infrared + elevation)</li>
    <li>model architecture : <b>resnet34_unet</b> (U-Net with a Resnet-34 encoder), <b>deeplab</b>, <b>fpn</b>, <b>mit</b></li>
    <li>target class nomenclature : <b>12cl</b> (12 land cover classes) or <b>15cl</b> (15 land cover classes)</li>
  </ul>
</div>
<br>

<div style="border:1px solid black; padding:25px; background-color:#FDFFF4 ; padding-top:10px; padding-bottom:1px;">
  <h1>FLAIR-INC_rgbi_15cl_resnet34-unet</h1> 
  <p>The general characteristics of this specific model <strong>FLAIR-INC_rgbi_15cl_resnet34-unet</strong> are :</p>
  <ul style="list-style-type:disc;">
    <li>Trained with the FLAIR-INC dataset</li>
    <li>Trained with the SegmentationModelsPytorch library</li>  
    <li>RGBI images (true colours + infrared )</li>
    <li>U-Net with a Resnet-34 encoder</li>
    <li>15 class nomenclature : [building, pervious surface, impervious surface, bare soil, water, coniferous, deciduous, brushwood, vineyard, herbaceous, agricultural land, plowed land, swimming pool, snow, greenhouse]</li>
  </ul>
</div>

## Model Informations
- **Code repository:** https://github.com/IGNF/FLAIR-1
- **Paper:** https://arxiv.org/pdf/2211.12979.pdf
- **Developed by:** IGN
- **Compute infrastructure:** 
    - software: python, pytorch-lightning
    - hardware: HPC/AI resources provided by GENCI-IDRIS
- **License:** Etalab 2.0

---

## Uses

Although the model can be applied to other type of very high spatial earth observation images, it was initially developed to tackle the problem of classifying aerial images acquired on the French Territory.
The product called ([BD ORTHO®](https://geoservices.ign.fr/bdortho)) has its own spatial and radiometric specifications. The model is not intended to be generic to other type of very high spatial resolution images but specific to BD ORTHO images. 
Consequently, the model’s prediction would improve if the user images are similar to the original ones.

_**Radiometry of input images**_ :
The BD ORTHO input images are distributed in 8-bit encoding format per channel. When traning the model, input normalization was performed (see section **Training Details**). 
It is recommended that the user apply the same type of input normalization while inferring the model.

_**Multi-domain model**_ :
The FLAIR-INC dataset that was used for training is composed of 75 radiometric domains. In the case of aerial images, domain shifts are frequent and are mainly due to : the date of acquisition of the aerial survey (from april to november), the spatial domain (equivalent to a french department administrative division) and downstream radiometric processing.
By construction (sampling 75 domains) the model is robust to these shifts, and can be applied to any images of the ([BD ORTHO® product](https://geoservices.ign.fr/bdortho)).

_**Land Cover classes of prediction**_ :
The orginial class nomenclature of the FLAIR Dataset encompasses 19 classes (See the [FLAIR dataset](https://huggingface.co/datasets/IGNF/FLAIR) page for details).
However 3 classes corresponding to uncertain labelisation (Mixed (16), Ligneous (17) and Other (19)) and 1 class with very poor labelling (Clear cut (15)) were desactivated during training.
As a result, the logits produced by the model are of size 19x1, but classes n° 15, 16, 17 and 19 should appear at 0 in the logits and should not be present in the final argmax product.



## Bias, Risks, Limitations and Recommendations

_**Using the model on input images with other spatial resolution**_ :
The FLAIR-INC_rgbi_15cl_resnet34-unet model was trained with fixed scale conditions. All patches used for training are derived from aerial images with 0.2 meters spatial resolution. Only flip and rotate augmentations were performed during the training process.  
No data augmentation method concerning scale change was used during training. The user should pay attention that generalization issues can occur while applying this model to images that have different spatial resolutions.

_**Using the model for other remote sensing sensors**_ :
The FLAIR-INC_rgbi_15cl_resnet34-unet model was trained with aerial images of the ([BD ORTHO® product](https://geoservices.ign.fr/bdortho)) that encopass very specific radiometric image processing. 
Using the model on other type of aerial images or satellite images may imply the use of transfer learning or domain adaptation techniques.

_**Using the model on other spatial areas**_ :
The FFLAIR-INC_rgbi_15cl_resnet34-unet model was trained on patches reprensenting the French Metropolitan territory. 
The user should be aware that applying the model to other type of landscapes may imply a drop in model metrics.  

---

## How to Get Started with the Model

Visit ([https://github.com/IGNF/FLAIR-1](https://github.com/IGNF/FLAIR-1)) to use the model.
Fine-tuning and prediction tasks are detailed in the README file.


---

## Training Details

### Training Data

218 400 patches of 512 x 512 pixels were used to train the **FLAIR-INC_rgbi_15cl_resnet34-unet** model. 
The train/validation split was performed patchwise to obtain a 80% / 20% distribution between train and validation. 
Annotation was performed at the _zone_ level (~100 patches per _zone_). Spatial independancy between patches is guaranted as patches from the same _zone_ were assigned to the same set (TRAIN or VALIDATION). 
The following number of patches were used for train and validation :
| TRAIN set            | 174 700 patches    |
| VALIDATION set       | 43 700 patchs      |





### Training Procedure

#### Preprocessing

For traning the model, input normalization was performed to center-reduce (**a mean=0** and a **standard deviation = 1**, channel wise) the dataset.
We used the statistics of TRAIN+VALIDATION for input normalization. It is recommended that the user apply the same type of input normalization. 

Statistics of the TRAIN+VALIDATION set :

| Modalities              | Mean (Train + Validation)       |Std    (Train + Validation)     |
| ----------------------- | ----------- |----------- |
| Red Channel (R)         | 105.08	    |52.17       |
| Green Channel (G)       | 110.87      |45.38       |
| Blue Channel (B)        | 101.82	    |44.00       |
| Infrared Channel (I)    | 106.38	    |39.69       |


#### Training Hyperparameters

```yaml
- Model architecture: Unet #(implementation from the [Segmentation Models Pytorch library](https://segmentation-modelspytorch.readthedocs.io/en/latest/docs/api.html#unet))
- Encoder : Resnet-34 pre-trained with ImageNet
- Augmentation :
  - VerticalFlip(p=0.5)
  - HorizontalFlip(p=0.5)
  - RandomRotate90(p=0.5)
- Input normalization (mean=0 | std=1):
  - norm_means: [105.08, 110.87, 101.82, 106.38]
  - norm_stds: [52.17, 45.38, 44, 39.69]
- Seed: 2022
- Batch size: 10
- Number of epochs : 200
- Early stopping : patience 30 and val_loss as monitor criterium 
- Optimizer : SGD
- Schaeduler : mode = "min", factor = 0.5, patience = 10, cooldown = 4, min_lr = 1e-7
- Learning rate : 0.02
- Class Weights : [1-building: 1.0 , 2-pervious surface: 1.0 , 3-impervious surface: 1.0 , 4-bare soil: 1.0 , 5-water: 1.0 , 6-coniferous: 1.0 , 7-deciduous: 1.0 , 8-brushwood: 1.0 , 9-vineyard: 1.0 , 10-herbaceous vegetation: 1.0 , 11-agricultural land: 1.0 , 12-plowed land: 1.0 , 13-swimming_pool: 1.0 , 14-snow: 1.0 , 15-clear cut: 0.0 , 16-mixed: 0.0 , 17-ligneous: 0.0 , 18-greenhouse: 1.0 , 19-other: 0.0]
```

#### Speeds, Sizes, Times

The FLAIR-INC_rgbi_15cl_resnet34-unet model was trained on a HPC/AI resources provided by GENCI-IDRIS (Grant 2022-A0131013803). 
16 V100 GPUs were used ( 4 nodes, 4 GPUS per node). With this configuration the approximate learning time is 6 minutes per epoch.

FLAIR-INC_rgbi_15cl_resnet34-unet was obtained for num_epoch=65 with corresponding val_loss=0.56. 


<div style="position: relative; text-align: center;">
    <p style="margin: 0;">TRAIN loss</p>
    <img src="FLAIR-INC_rgbi_15cl_resnet34-unet_train-loss.png" alt="TRAIN loss" style="width: 60%; display: block; margin: 0 auto;"/>
    <p style="margin: 0;">VALIDATION loss</p>
    <img src="FLAIR-INC_rgbi_15cl_resnet34-unet_val-loss.png" alt="VALIDATION loss" style="width: 60%; display: block; margin: 0 auto;"/>
</div>



## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

The evaluation was performed on a TEST set of 31 750 patches that are independant from the TRAIN and VALIDATION patches. They represent 15 spatio-temporal domains.
The TEST set corresponds to the reunion of the TEST set of scientific challenges FLAIR#1 and FLAIR#2. See the [FLAIR challenge page](https://ignf.github.io/FLAIR/) for more details.

The choice of a separate TEST set instead of cross validation was made to be coherent with the FLAIR challenges. 
However the metrics for the Challenge were calculated on 12 classes and the TEST set acordingly. 
As a result the _Snow_ class is absent from the TEST set.

#### Metrics

With the evaluation protocol, the **FLAIR-INC_rgbi_15cl_resnet34-unet** have been evaluated to **OA= 76.259%** and **mIoU=60.718%**. 
The _snow_ class is discarded from the average metrics.

The following table give the class-wise metrics :

  | Classes               |   IoU (%)    | Fscore (%)  | Precision (%)  | Recall (%)  |
| ----------------------- | ----------|---------|---------|---------|
|  building   		      |   78.614   |   88.027   |   88.596   |   87.465   |
|  pervious_surface       |   52.657   |   68.988   |   70.900   |   67.176   |
|  impervious_surface     |   72.566   |   84.102   |   84.002   |   84.203   |
|  bare_soil   	          |   59.656   |   74.731   |   77.464   |   72.184   |
|  water   		          |   87.604   |   93.393   |   92.367   |   94.442   |
|  coniferous   	      |   62.255   |   76.737   |   77.340   |   76.144   |
|  deciduous   	          |   71.762   |   83.560   |   81.669   |   85.540   |
|  brushwood   	          |   31.270   |   47.643   |   59.508   |   39.723   |
|  vineyard   		      |   76.454   |   86.656   |   85.320   |   88.034   |
|  herbaceous   	      |   51.643   |   68.111   |   70.792   |   65.625   |
|  agricultural_land      |   57.639   |   73.127   |   66.930   |   80.590   |
|  plowed_land   	      |   43.452   |   60.581   |   58.839   |   62.430   |
|  swimming_pool   	      |   41.260   |   58.417   |   79.497   |   46.173   |
|  snow   		          |   _0.000_  |   _0.000_   |   _0.000_   |   _0.000_   |
|  greenhouse   	      |   63.222   |   77.467   |   69.990   |   86.733   |
| **average**             |  **60.718**    |  **74.396**  |  **75.944**  |  **74.033**  |








The following illustration gives the resulting confusion matrix :
* Top : normalised acording to columns, columns sum at 100% and the **precision** is on the diagonal of the matrix
* Bottom : normalised acording to rows, rows sum at 100% and the **recall** is on the diagonal of the matrix 


<div style="position: relative; text-align: center;">
    <p style="margin: 0;">Normalized Confusion Matrix (precision)</p>
    <img src="FLAIR-INC_rgbi_15cl_resnet34-unet_confmat_norm-precision.png" alt="drawing" style="width: 70%; display: block; margin: 0 auto;"/>
    <p style="margin: 0;">Normalized Confusion Matrix (recall)</p>
    <img src="FLAIR-INC_rgbi_15cl_resnet34-unet_confmat_norm-recall.png" alt="drawing" style="width: 70%; display: block; margin: 0 auto;"/>
</div>



### Results

Samples of results


---

## Citation


**BibTeX:**

```
@inproceedings{ign-flair,
      title={FLAIR: a Country-Scale Land Cover Semantic Segmentation Dataset From Multi-Source Optical Imagery}, 
      author={Anatol Garioud and Nicolas Gonthier and Loic Landrieu and Apolline De Wit and Marion Valette and Marc Poupée and Sébastien Giordano and Boris Wattrelos},
      year={2023},
      booktitle={Advances in Neural Information Processing Systems (NeurIPS) 2023},
      doi={https://doi.org/10.48550/arXiv.2310.13336},
}
```


**APA:**
```
Anatol Garioud, Nicolas Gonthier, Loic Landrieu, Apolline De Wit, Marion Valette, Marc Poupée, Sébastien Giordano and Boris Wattrelos. 2023. 
FLAIR: a Country-Scale Land Cover Semantic Segmentation Dataset From Multi-Source Optical Imagery. (2023).
In proceedings of Advances in Neural Information Processing Systems (NeurIPS) 2023.
DOI: https://doi.org/10.48550/arXiv.2310.13336
```

## Contact : flair@ign.fr