OneScience-Group/TerraMind

TerraMind

Model Introduction

TerraMind is an any-to-any generative foundation model for Earth observation data. It jointly processes pixel-level and discrete token-level data and learns cross-modal relationships among radar, optical imagery, elevation, land cover, vegetation indices, geographic coordinates, and text.

Paper: TerraMind: Large-Scale Generative Multimodality for Earth Observation
https://arxiv.org/abs/2504.11171

Model Description

TerraMind was proposed by organizations including IBM Research, the European Space Agency, and Forschungszentrum Jülich. The model was pretrained on approximately 9 million globally distributed, spatiotemporally aligned TerraMesh samples and approximately 500 billion training tokens. It is suitable for cross-modal generation, Earth observation representation learning, land-cover segmentation, water-body identification, vegetation assessment, and Thinking-in-Modalities tasks.

Use Cases

Use Case	Description
Multimodal representation	Jointly encodes optical, radar, elevation, and other Earth observation modalities.
Any-to-any generation	Predicts target-modality tokens such as LULC, NDVI, and radar from available modalities such as Sentinel-2, coordinates, and text.
Dual-scale learning	Uses both raw pixel patches and discrete token representations.
Local engineering validation	Uses aligned synthetic data to verify the training, inference, evaluation, visualization, and checkpoint workflows.
Multi-GPU training	Launches distributed data-parallel training through torchrun.

Usage Instructions

1. OneCode

Experience intelligent, one-click AI4S programming through the OneCode online environment:

Try intelligent, one-click AI4S programming

2. Download and Installation

hf download OneScience-Group/TerraMind --local-dir ./TerraMind
cd TerraMind

Environment Dependencies

Hardware Requirements

• A GPU or DCU is recommended.
• A CPU can be used to verify connectivity with the default small-sample configuration; official-scale training and diffusion decoding require large-scale accelerated computing resources.
• DCU users must install DTK in advance. DTK 25.04.2 or later, or the OneScience-recommended version matching the current cluster, is recommended.

DCU Environment

# Activate DTK and Conda first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai

GPU Environment

# Activate Conda first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai

Training Data

This repository uses spatiotemporally aligned synthetic multimodal samples to validate the engineering workflow. The source data preserve TerraMesh's 264×264 spatial dimensions and include 12-channel Sentinel-2 L2A, 13-channel Sentinel-2 L1C, 3-channel RGB, 2-channel Sentinel-1 GRD, 2-channel Sentinel-1 RTC, and single-channel DEM data. The loader jointly crops the 224×224 inputs used by the official model from each source sample and constructs content-dependent discrete tokens for LULC, NDVI, radar, and other modalities from the same region, ensuring dual-scale spatial alignment.

python scripts/fake_data.py

Training

python scripts/train.py

For multi-GPU training, use:

torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py

The default configuration only reduces the number of samples, Transformer width, depth, vocabulary, and training epochs; it does not change the source-data or model-input spatial dimensions. During training, the workflow randomly selects input pixel modalities, input token modalities, and target modalities, and randomly samples input patches to simulate the paper's multimodal masked-modeling strategy.

result/checkpoints/terramind.pt
result/training/metrics.json

Trained Weights

This repository does not include synthetic or official weights in weight/. IBM and ESA have published model weights for TerraMind variants including tiny, small, base, and large.

Inference

python scripts/inference.py

Inference loads the training checkpoint and generates LULC, NDVI, and Sentinel-1 GRD tokens conditioned only on Sentinel-2 L2A pixels, coordinate tokens, and text tokens. It saves the cross-modal embeddings and target tokens to:

result/output/predictions.npz

Evaluation and Visualization

python scripts/result.py

The evaluation measures token accuracy for each target modality and cross-modal embedding norms, and generates spatial comparison plots of target and generated tokens. Results on synthetic data only validate the engineering pipeline and do not represent the PANGAEA, generation-quality, or Thinking-in-Modalities metrics reported in the paper.

result/evaluation/metrics.json
result/evaluation/comparison.png

Official OneScience Information

Platform	OneScience Main Repository	Skills Repository
Gitee	https://gitee.com/onescience-ai/onescience	https://gitee.com/onescience-ai/oneskills
GitHub	https://github.com/onescience-ai/OneScience	https://github.com/onescience-ai/oneskills

Citation and License

This repository is an independent engineering reproduction based on the publicly available specifications from the TerraMind paper.

Use of this repository's code, the official model weights, and the data remains subject to the licenses and terms of use of their respective projects.