Amandeep10/pretrain-where-SatMAE-weights

Model card for Pretrain-Where SatMAE weights

Pretrain-Where SatMAE weights

SatMAE (https://arxiv.org/abs/2207.08051) is the most cited MAE-based GeoFoundational Model. We used SatMAE as the base model for our work 'Pretrain Where? Investigating How Pretraining Data Diversity Impacts Geospatial Foundation Model Performance' (https://arxiv.org/abs/2604.21104). We pretrained SatMAE on 7 geographically diverse contexts - 6 continents (except Antarctica) and 1 global data set, and while keeping everything else constant, we assessed downstream model performance on a diverse set of downstream tasks. We compared these numbers to three pretraining baseline datasets as well, namely - FMoW-Sentinel, SSL4Eco, and SSL4Eo. We provide pretrained SatMAE weights for these 10 SatMAE variants (6 continents, 1 global, 3 baselines). We also independently experimented with pretraining on different-sized subsets of FMoW-Sentinel, leading to VitB-FMoW-100k, VitB-FMoW-300k, ..-500k, ..700k, and VitL-FMoW-700k, weights for which are also stored here.

Architecture

SatMAE is based on Vit-MAE architecture. We used the Vit-Base (unless stated otherwise) grouped-channel variant for the pretrain-where experiments.

Pretraining Details

Details used to pretrain SatMAE for pretrain-where experiments (also default to SatMAE):

• batch size = 256
• accum iter = 16
• blr = 0.0001
• epochs = 50
• warmup epochs = 5
• input size = 96
• patch size = 8
• mask ratio = 0.75
• model = ’mae_vit_base_patch16’
• model type = ’group_c’
• dropped_channels = 0 9 10
• grouped_channels = default of SatMAE

Datasets

The custom datasets used to pretrain the 7 geographically diverse SatMAE variants are available here: https://huggingface.co/datasets/Amandeep10/pretrain-where-pretraining-datasets

Checkpoint

We pre-trained all models for 50 epochs as done in the SatMAE paper. 49th-epochs (0-indexed) for each model variant are provided in this repo. Each checkpoint file (.pth) is a PyTorch pickle dict, typically including:

• model: state_dict for the MAE encoder (use this for finetuning)
• optimizer, epoch, args (argparse.Namespace), etc.

PyTorch ≥ 2.6: load with weights_only=False (checkpoints contain argparse.Namespace).

Intended use & Limitations

• Research / geospatial SSL and finetuning on satellite data.
• Not a deployed product; no warranty on geographic fairness or performance outside pretraining distribution.
• Large files (1.3 GB ViT-B, ~3.7 GB ViT-L); need disk and huggingface_hub.

Citation

Please cite our paper using:

  @misc{kaur2026pretrainwhereinvestigatingpretraining,
        title={Pretrain Where? Investigating How Pretraining Data Diversity Impacts Geospatial Foundation Model Performance}, 
        author={Amandeep Kaur and Mirali Purohit and Gedeon Muhawenayo and Esther Rolf and Hannah Kerner},
        year={2026},
        eprint={2604.21104},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2604.21104}, 
  }