FatimahEmadEldin/leo-uav-flair-experiments

LEO satellite + UAV image fusion — experiments

Everything for the paper in one place: 9 end-to-end Colab notebooks, the experiment tracker, and the canonical class definitions.

Start here: open LEO_UAV_FLAIR_Model_Tracker_v2.xlsx — 9 sheets covering both tasks, the folder map, the metric definitions, and the open questions.

Layout

LEO_UAV_FLAIR_Model_Tracker_v2.xlsx   the tracker (start here)
class_sets.json                       canonical label groupings
_shared_backbones.py                  unet / resnet / convnext / swin
make_tracker_v2.py                    refresh the tracker from run logs

segmentation/                         TASK 1
  01_baseline_single_stream/        The floor and the ceiling
  02_early_fusion_concat/           FusionHandler — the control
  03_crossscale_attention/          Cross-scale attention — the proposal
  04_backbone_resnet/               Same fusion, RESIDUAL encoder
  05_backbone_convnext/             Same fusion, CONVNEXT encoder
  06_backbone_swin/                 Same fusion, SWIN TRANSFORMER encoder
  07_class_set_ablation/            Where does fusion's value actually come from?

resolution_enhancement/               TASK 2
  01_s2_to_spot_l1/                 Sentinel-2 → SPOT (primary) (6.4x)
  02_spot_to_aerial_l1/             SPOT → aerial (4x)
  03_s2_to_aerial_l1/               Sentinel-2 → aerial (25.6x)
  04_backbone_rcan/                 Same regime, CHANNEL-ATTENTION model (6.4x)
  05_backbone_swinir/               Same regime, TRANSFORMER model (6.4x)
  06_conditional_diffusion/         Same regime, DIFFUSION loss family (6.4x)

Each methodology folder holds its notebook and a README explaining what that experiment answers, the exact configuration baked in, and where its results land.

Optimised for a Tesla T4

Every notebook defaults to PACK = "t4" — the 128 px pack, 0.90 GB, all 9 domains and all labels, a quarter of the pixels per training step. Switch to PACK = "full" for 256 px when you have an A100.

Environment	Works?	Note
Colab free (T4, 12.7 GB RAM)	yes	the default configuration
Colab Pro (A100 / L4)	yes	use PACK = "full"
Local, 16 GB RAM + 6 GB GPU	yes
Local, 8 GB RAM	careful	reduce shards to 3–4 domains
CPU only	no	~40 h per run

An earlier version decoded the whole dataset into RAM (~7 GB, peaking higher on np.stack) and OOM-killed a free T4. Every notebook now streams shards into memory-mapped files on local SSD, so peak RAM is one batch regardless of dataset size. If a notebook prints loaded N tiles into RAM, you have a cached old copy — disconnect and delete the runtime, then reopen.

On reducing the number of classes

Reducing classes does not reduce memory or download size. Labels are 2.3% of the data (1.7 KB per tile) and are stored as raw COSIA 0–18; the class set is a training-time lookup table in class_sets.json. Four classes and six classes occupy identical bytes. What reduces the footprint is patch size (PACK) and domain count.

Three sets ship, and the choice is an experimental variable:

set	classes	purpose
environment6	6	the headline set
forestmerged5	5	isolates the Deciduous+Coniferous merge
thematic4	4	coarse land-cover grouping

Measured per-class fusion gains: Coniferous +26.4 IoU, Water +15.3, Bare soil +10.2, Herbaceous +6.3, Deciduous +3.7, Agricultural −1.6. Conifer and broadleaf look alike in summer RGB and are separated by red-edge and SWIR that only the satellite carries — so thematic4, which merges them, dissolves the confusion the satellite is resolving.

That makes it a useful experiment (see 05_class_set_ablation) and a misleading headline. It scores higher because the task is easier, not because the model is better.

Data

Pack	Patch	Size	Use
flair-multidomain-parquet-128	128 px	0.90 GB	default, T4
flair-multidomain-parquet	256 px	1.94 GB	headline runs

24,475 tiles · 9 French domains · 257+ sectors, repacked from IGNF/FLAIR-HUB. RGB is JPEG q95, measured at −0.03 mIoU (0.01 σ) versus lossless. Labels bit-exact. Sentinel-2 bit-exact after zstd, 12 composites from 38–73 dates with >50% cloud-probability dates dropped.

Checkpoints and per-run documentation: leo-uav-fusion-checkpoints

The benchmark

FLAIR-HUB (arXiv:2506.07080) gains +0.6 mIoU from adding Sentinel-2 to aerial (LC-A 64.1 → LC-D 64.7) using 89.4 M parameters, 152,225 patches and ~20 V100/A100/H100 GPUs. Their fusion is concatenation after upsampling.

+0.6 is the number to beat — not their 64.1, which this hardware cannot reach and which we do not claim. That makes the contribution a statement about the fusion mechanism at matched budget, which is reachable on a T4.