SelenoVLM

SelenoVLM is a vision-language model for lunar geology. A multimodal masked autoencoder (MAE) trained over co-registered global lunar rasters (LROC WAC imagery, LOLA/GLD100 elevation, the Unified Geologic Map of the Moon, Clementine UVVIS color-ratio composite) serves as a frozen vision tower, coupled to a LoRA-adapted Qwen2.5-7B-Instruct through a Perceiver resampler and Flamingo-style gated cross-attention. The model writes natural-prose geological interpretations of 51 x 67 km lunar surface tiles: units, processes, relative timing, quantitative image facts, and crater-count model ages on mare surfaces.

Code, training pipeline, and evaluation: github.com/TechnicToms/SelenoVLM

Repository contents

Path	Content	Size
bridge/step-10000.ckpt	trained bridge (Perceiver resampler + gated cross-attention + LoRA), trainable params only	1.6 GB
mae/best-epoch10-step21428.ckpt	frozen multimodal MAE vision tower	3.2 GB
tokenizers/{wac,elevation,clementine,geomap}/	FSQ tokenizer checkpoints + hparams	260 MB
extras/UnifiedGeoMap/legend.pkl	geologic-map legend (required at build time)	2 KB
demo/tile_*/	preprocessed held-out demo tiles (inputs, WAC preview, metadata)	117 MB

The Qwen2.5-7B-Instruct base weights are not included; they are pulled from the HF hub on first build.

Usage

Inference goes through the GitHub repository (the checkpoints depend on its model code):

git clone https://github.com/TechnicToms/SelenoVLM && cd SelenoVLM
uv sync
python download_weights.py   # downloads this repository into the expected layout
python demo.py               # closed-book interpretation of a bundled held-out tile

Notes:

• Loading the bridge checkpoint reports roughly 1800 missing keys and 0 unexpected keys. This is expected: the checkpoint stores only the trainable bridge/LoRA parameters, and the frozen MAE and base LLM are rebuilt at load time.
• Requires a CUDA GPU with about 24 GB VRAM (7B LLM in bf16).
• The demo tiles are from a spatially disjoint held-out test split, selected among the tiles where the served checkpoint performs well; they showcase the intended behaviour (grounded numeric image facts, mare age statements, highland age silence).

Training summary

• Vision tower: shared multimodal MAE (4M-style architecture) over five modalities at 100 m/px, 512 px tiles, trained with Dirichlet span masking. Warm-started from EPFL/Apple 4M weights, fully retrained on lunar data.
• Bridge: only the resampler, gated cross-attention blocks (every 4th decoder layer, no FFN), and LoRA adapters train (~780 M params, 2 epochs). Supervised targets are RAG-grounded Hiesinger-style prose with deterministic per-tile image facts injected; prompts and targets are coordinate-free.
• Checkpoint selection: by causal vision grounding, grounded-fact fidelity, age-emission behaviour, and degeneration metrics on held-out tiles (not by validation loss).

Evaluation

Closed-book evaluation on the spatially disjoint test split (see the GitHub repository for the full protocol and reports): grounded numeric image facts scored against the tile rasters, mare-gated age emission with highland silence, causal vision ablations (zero and shuffle flips), and an always-on constant-age baseline guard. Numeric age values on mare tiles track the regional/global mare prior; the model's dating skill is in emission behaviour and epoch vocabulary, not in beating a constant-age predictor. Treat stated numeric ages accordingly.

License and provenance

Released under Apache-2.0. Upstream terms that apply in addition:

• Base LLM: Qwen2.5-7B-Instruct (Apache-2.0), not redistributed here.
• The MAE was initialized from EPFL/Apple 4M weights and fully retrained; see the ml-4m repository for their model license.
• The supervised prose targets were distilled with Gemma 3; use of this model is additionally subject to the Gemma Terms of Use.
• Lunar data products are NASA/USGS public domain; Hiesinger mare ages derive from published literature, used with attribution.

Citation

A preprint is in preparation; citation information will be added here.