BiliSakura/SATMAE-PP-transformers

SatMAE++ Transformers Models

Hugging Face–compatible checkpoints converted from the official SatMAE++ pretrained weights. Each subfolder is a standalone model repo layout (config.json, model.safetensors, preprocessor, and remote code) for feature extraction on FMoW satellite imagery.

Model Description

These models are ViT-Large encoders pretrained with multi-scale masked autoencoding (SatMAE++) on fMoW-RGB and fMoW-Sentinel imagery.

This collection currently bundles 2 converted pretrain checkpoints:

• FMoW-RGB: vanilla ViT-L/16, 3-channel BGR, 224×224
• FMoW-Sentinel: grouped-channel ViT-L/8, 10-band multispectral, 96×96

All folders ship self-contained remote code (modeling_satmae_pp.py, processor, pipeline) and load with trust_remote_code=True.

Developed by: techmn / SatMAE++
Converted for Hugging Face by: BiliSakura
License (weights): MIT
Original paper: Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery (CVPR 2024)

Available checkpoints

Folder	Dataset	Encoder	Channels	Image	Patch	Legacy file
satmae-pp-vit-large-patch16-fmow-rgb-pretrain	FMoW-RGB	vanilla ViT	3 (BGR)	224	16	checkpoint_ViT-L_pretrain_fmow_rgb.pth
satmae-pp-vit-large-patch8-fmow-sentinel-pretrain	FMoW-Sentinel	group-channel ViT	10	96	8	checkpoint_ViT-L_pretrain_fmow_sentinel.pth

Legacy .pth filename mapping is in conversion_manifest.json.

Usage

Processors default to do_resize: false. Inputs keep native height and width; FMoW mean/std normalization still applies when enabled.

from transformers import pipeline
import numpy as np

REPO = "/path/to/SATMAE-PP-transformers"
SUBFOLDER = "satmae-pp-vit-large-patch8-fmow-sentinel-pretrain"

pipe = pipeline(
    task="satmae-pp-feature-extraction",
    model=REPO,
    trust_remote_code=True,
    model_kwargs={"subfolder": SUBFOLDER},
)

# FMoW-Sentinel: 10 bands at native size (e.g. 96×96 or larger)
image = np.random.randint(0, 255, (128, 128, 10), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # [1, 1024]

FMoW-RGB (BGR channel order):

SUBFOLDER = "satmae-pp-vit-large-patch16-fmow-rgb-pretrain"
pipe = pipeline(
    task="satmae-pp-feature-extraction",
    model=REPO,
    trust_remote_code=True,
    model_kwargs={"subfolder": SUBFOLDER},
)
image = np.random.randint(0, 255, (384, 384, 3), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # [1, 1024]

To resize to the pretraining reference (96×96 sentinel / 224×224 RGB):

features = pipe(image, pool=True, return_tensors=True, image_processor_kwargs={"do_resize": True})

Load components directly:

from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)

Normalization

The bundled image processor applies per-channel FMoW mean/std normalization by default (do_normalize=True). FMoW-RGB models expect BGR channel order; the processor swaps RGB→BGR when channel_order="bgr".

For FMoW-Sentinel, inputs should be 10-band reflectance arrays (bands 0, 9, 10 already dropped) with FMoW-Sentinel statistics baked into the preprocessor config.

Dependencies

• transformers, torch, timm, safetensors
• opencv-python (multispectral resize with more than 4 channels)

Citation

@inproceedings{satmaepp2024rethinking,
  title={Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery},
  author={Mubashir Noman and Muzammal Naseer and Hisham Cholakkal and Rao Muhammad Anwar and Salman Khan and Fahad Shahbaz Khan},
  year={2024},
  booktitle={CVPR}
}