SpaceSense-Omni

Unified RGB/LiDAR spacecraft perception, initialized from 4M-21-B. This release uses the fixed 1024 adaptation best checkpoint (epoch 4, step 6472), selected by the equal-weight five-task validation score. It does not combine task-specific weights. See release.json for the source checksum, architecture and calibration.

RGB + LiDAR supports eight-class pixel segmentation (background plus seven parts), seven-class FCOS detection, seven-class original-point segmentation, metric ray distance, and relative pose. RGB-only returns the four image/pose tasks and marks point segmentation unavailable. It is experimental: training used lidar_dropout=0.

Usage

Download the complete model repository, install the Space requirements, add vendor and the repository root to PYTHONPATH, then:

from spacesense.hf_runtime import Predictor
predictor = Predictor.from_bundle('/path/to/model-repository', device='cuda')
prediction, metadata = predictor.predict('/path/to/image.png')
# Fusion also takes N x 3 sensor-frame FRD points (metres) and the image's K:
# prediction, metadata = predictor.predict(image, points, intrinsics)

The release contains FP32 safetensors and configs, without optimizer or unused 4M decoder. GPU inference uses BF16 autocast; the pose head remains FP32. Images are resized to 1024 x 1024; pixel outputs are resized back to input dimensions. LiDAR calibration applies exactly once: sensor FRD (x,y,z) -> optical (y,z,x+0.9). The calibrated sensor configuration is specific to the supplied examples. Detection classes are 0..6; image/point foreground labels are 1..7; point label 0 is unknown.

Evaluation and limitations

EVALUATION.md reports paired RGB/fusion results on 680 predetermined demonstration frames. These are not full-test-set results. ID contains 117 training satellites; OOD contains 5 validation satellites used for selection and 14 test satellites. No new training or tuning on these examples was performed.

Synthetic spacecraft data, assumed camera geometry and lidar_dropout=0 limit generalization to arbitrary uploaded real images. In particular, absolute depth and translation must not be interpreted as calibrated measurements of arbitrary images. Training lateral translation was zero. Small-part recognition can fail. Point truth is generated by 2D/depth projection, not independent manual 3D labels. In the fixed 70-frame test-satellite demo subset, removing LiDAR increased median translation error from approximately 0.21 m to 29.00 m and metric-depth AbsRel from 0.056 to 2.53. RGB-only depth and translation are therefore not reliable metric measurements. See EVALUATION.md for the complete paired results.

Attribution and license

Derived from EPFL-VILAB/4M-21_B and the 4M/4M-21 work of Mizrahi et al. and Bachmann et al. https://4m.epfl.ch/ . Model weights retain the upstream Sample Code License, which permits non-commercial use. See LICENSE. Upstream Python code has separate Apache-2.0 and component notices, retained under vendor/. This release adds the SpaceSense task heads, adapted encoder weights, inference interface and demo.