OneScience-Group/SkySense

SkySense

Model Introduction

SkySense is a multi-modal and temporal remote sensing foundation model for universal Earth observation interpretation. It jointly models high-resolution optical imagery, Sentinel-2 multispectral time series, Sentinel-1 SAR time series, and geographic context to provide unified representations for remote sensing classification, segmentation, detection, and change detection.

Paper: SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery

https://arxiv.org/abs/2312.10115

Model Description

SkySense was proposed by Ant Group, Wuhan University, and MYbank. The original model was trained on 21.5 million groups of high-resolution optical, Sentinel-2 multispectral, and Sentinel-1 SAR temporal samples, and is suitable for single-modal or multi-modal, static or temporal remote sensing classification and localization tasks.

Use Cases

Scenario	Description
Multi-modal remote sensing representation training	Train SkySense with multi-modal temporal remote sensing image data.
Remote sensing image semantic segmentation	Fuse optical, multispectral, and SAR features to predict land-cover classes for image pixels.
Local quick validation	Use synthetic data to validate data loading, training, inference, evaluation, and visualization.
Hugging Face / OneCode execution	Download the standalone model package, install dependencies, and run the scripts directly.
Multi-GPU training	Launch distributed multi-process training with torchrun.

Usage Guide

1. OneCode Usage

Experience intelligent one-click AI4S programming through the OneCode online environment:

Click to Experience Intelligent One-Click AI4S Programming

2. Manual Installation and Usage

Hardware Requirements

• A GPU or DCU is recommended.
• CPU can be used for import and small-scale connectivity verification; full training and inference will be slower.
• DCU users must install DTK in advance. DTK 25.04.2 or above, or the OneScience recommended version matching the cluster, is recommended.

Download the Model Package

hf download OneScience-Group/SkySense --local-dir ./SkySense
cd SkySense

Install the Runtime Environment

DCU Environment

# Please activate DTK and CONDA first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
# uv installation is supported
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai

GPU Environment

# Please activate CONDA first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
# uv installation is supported
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai

Training Data Introduction

The original SkySense pretraining collection contains about 21.5 million spatially registered multi-modal remote sensing samples. It includes high-resolution optical imagery, Sentinel-2 multispectral temporal imagery, Sentinel-1 SAR temporal imagery, acquisition dates, geographic region information, and downstream task labels. The complete registered pretraining collection is not publicly available from the authors, and it is not included in this model repository.

Original data sources include Sentinel-1 and Sentinel-2 products available through Copernicus Data Space and commercial high-resolution imagery such as WorldView. Users must follow the applicable data licenses and independently complete cloud filtering, radiometric processing, spatial registration, resampling, cropping, normalization, and label preparation.

The default local validation uses synthetic NPZ data:

python scripts/fake_data.py

The command creates 24 training samples and 8 test samples. Synthetic data is used only to validate the complete training, inference, evaluation, and visualization pipeline. Synthetic results are not reported as model performance and do not reproduce the paper experiments.

For real-data training, prepare the corresponding multi-modal samples in the format expected by the scripts, save them under data/, update the data and model settings in conf/config.yaml, and use the same scripts/train.py. Do not run scripts/fake_data.py when using real data.

Training

Synthetic-data quick validation:

python scripts/fake_data.py
python scripts/train.py

Single GPU or CPU:

python scripts/train.py

Multi-GPU:

torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py

Training outputs:

result/checkpoints/skysense.pt
result/training/metrics.json

Training Weights

Pretrained weights based on multi-modal temporal remote sensing data will be provided under weight/. The weight file will be uploaded in a future update. Checkpoints generated from synthetic data are only for pipeline validation and do not provide real remote sensing interpretation capability.

Inference

python scripts/inference.py

Inference reads data/test.npz and result/checkpoints/skysense.pt. Test data must use the same channel order, normalization, spatial size, region mapping, and class definition as the training data.

Prediction output:

result/output/

Evaluation and Visualization

python scripts/result.py

Evaluation and visualization outputs:

result/evaluation/metrics.json
result/evaluation/comparison.png

For synthetic data, evaluation outputs only confirm that the evaluation and visualization pipeline can run successfully. They are not reported as model performance. For real data, the metrics represent supervised semantic segmentation performance on the user-provided test set and are not equivalent to the complete 21.5-million-sample pretraining evaluation in the paper.

Official OneScience Resources

Platform	OneScience Main Repository	Skills Repository
Gitee	https://gitee.com/onescience-ai/onescience	https://gitee.com/onescience-ai/oneskills
GitHub	https://github.com/onescience-ai/OneScience	https://github.com/onescience-ai/oneskills

Citation and License

• This repository is a reproduction of the original SkySense paper.
• The reproduction code is released under Apache License 2.0. The original datasets and official model weights remain subject to their respective source licenses.