OneScience-Group/SpectralGPT

SpectralGPT

Model Introduction

SpectralGPT learns general-purpose representations from large-scale multispectral remote sensing imagery through three-dimensional spatial-spectral masked modeling and multi-objective reconstruction. It is designed for scene classification, semantic segmentation, and change detection under limited-label conditions.

Paper: SpectralGPT: Spectral Remote Sensing Foundation Model

https://arxiv.org/abs/2311.07113

Model Description

SpectralGPT was proposed by researchers from the Aerospace Information Research Institute, Chinese Academy of Sciences, and related institutions. It is trained with 12-band Sentinel-2 imagery from fMoW-S2 and BigEarthNet-S2 and is suitable for single-label or multi-label scene classification, semantic segmentation, and change detection.

Use Cases

Scenario	Description
Multispectral masked pretraining	Train SpectralGPT with normalized 12-band Sentinel-2 multispectral imagery.
Remote sensing scene understanding	Use learned multispectral representations for scene classification, semantic segmentation, and change detection.
Local quick validation	Use synthetic data to validate data loading, masked reconstruction training, inference, evaluation, and visualization.
Hugging Face / OneCode execution	Download the standalone model package, install dependencies, and run the scripts directly.
Multi-GPU training	Launch distributed multi-process training with torchrun.

Usage Guide

1. OneCode Usage

Experience intelligent one-click AI4S programming through the OneCode online environment:

Click to Experience Intelligent One-Click AI4S Programming

2. Manual Installation and Usage

Hardware Requirements

• A GPU or DCU is recommended.
• CPU can be used for import and small-scale connectivity verification; full training and inference will be slower.
• DCU users must install DTK in advance. DTK 25.04.2 or above, or the OneScience recommended version matching the cluster, is recommended.

Download the Model Package

hf download OneScience-Group/SpectralGPT --local-dir ./SpectralGPT
cd SpectralGPT

Install the Runtime Environment

DCU Environment

# Please activate DTK and CONDA first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
# uv installation is supported
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai

GPU Environment

# Please activate CONDA first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
# uv installation is supported
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai

Training Data Introduction

The paper uses 12 Sentinel-2 spectral bands and excludes B10, with each band scaled to [0,1]. The fMoW-S2 dataset contains 882,779 images, including 712,874 images used for the first pretraining stage. BigEarthNet-S2 contains 590,326 images, including 354,196 images used for subsequent pretraining. OneScience does not currently provide fMoW-S2, BigEarthNet-S2, or EuroSAT data that can be downloaded directly for this repository, so the default workflow uses a small synthetic multispectral dataset for pipeline validation.

Official EuroSAT data resources:

Official Zenodo page:
https://zenodo.org/records/7711810

Multispectral data download:
https://madm.dfki.de/files/sentinel/EuroSATallBands.zip

After extracting the EuroSAT multispectral data, use the following command to exclude B10, scale the bands to [0,1], and convert the 13-band TIFF files into the 12-band NPZ data used by this repository:

python scripts/fake_data.py --real-dir ./data/EuroSAT_MS

Synthetic data is saved to data/fake_spectralgpt.npz and is used only to validate data loading, masked reconstruction training, checkpoint saving and loading, inference, evaluation, and visualization. It does not represent real Sentinel-2 imagery and cannot reproduce the paper results. Real-data training should not run scripts/fake_data.py without --real-dir, and the input TIFF files must use a consistent band order, spatial size, and value scaling method.

Training

Local quick validation with synthetic data:

python scripts/fake_data.py
python scripts/train.py

Training with real EuroSAT multispectral TIFF files:

python scripts/fake_data.py --real-dir ./data/EuroSAT_MS
python scripts/train.py

Synthetic and real data use the same training script. Adjust data.samples, model.image_size, training.epochs, and training.batch_size in conf/config.yaml as needed.

Single GPU or CPU:

python scripts/train.py

Multi-GPU:

torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py

Training outputs:

result/checkpoints/best.pth
result/checkpoints/last.pth
Each epoch prints reconstruction_loss.

The data_source and protocol fields in the training logs and checkpoints identify whether synthetic or real data was used. Checkpoints generated from synthetic data are only for pipeline validation and do not provide real remote sensing representations.

Training Weights

Multispectral Sentinel-2 training weights will be provided under weight/ in a future update. No trained weights are included in the current package.

Inference

python scripts/inference.py

Inference reads the configured dataset and the checkpoint generated during training, then produces multispectral image reconstruction results.

Prediction output:

result/output/reconstruction.npz

Evaluation and Visualization

python scripts/result.py

Evaluation and visualization outputs:

result/output/metrics.json
result/output/reconstruction.ppm

The metrics include reconstruction MSE, MAE, PSNR, and per-band RMSE. The visualization displays input and reconstructed false-color composites. Synthetic data is used only to confirm that the evaluation and visualization pipeline runs successfully; its numerical values are not reported as model performance. Real-data metrics represent masked reconstruction on the user-provided imagery.

Official OneScience Resources

Platform	OneScience Main Repository	Skills Repository
Gitee	https://gitee.com/onescience-ai/onescience	https://gitee.com/onescience-ai/oneskills
GitHub	https://github.com/onescience-ai/OneScience	https://github.com/onescience-ai/oneskills

Citation and License

• This repository is a reproduction of the original SpectralGPT paper.
• The model package is released under Apache License 2.0. The original datasets and model weights remain subject to their respective source licenses.