Probing-VLM-VGM Checkpoints

Paper | GitHub

Official trained probe checkpoints for Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models.

About

This repository provides 54 trained probes covering 18 foundation models and three tasks: semantic tagging and instance grouping on ScanNet, and 3D geometry prediction on DL3DV. The probes read frozen VLM or VGM features to assess their semantic, object-level, and geometric information.

These checkpoints contain the probe weights, not the frozen foundation-model weights. They are compact Lightning checkpoints containing the model state dictionary and Lightning version, without optimizer or other resumable training state. Datasets and extracted features are not included.

For installation, data preparation, feature extraction, and download/evaluation commands, see the GitHub README.

Repository Layout

.
|-- README.md
|-- manifest.json
|-- semantic_tagging/
|   `-- <model>/
|       |-- probe.ckpt
|       |-- probe_config.json
|       |-- metadata.json
|       |-- clip_class_embeds.npy
|       `-- class_names.json
|-- instance_grouping/
|   `-- <model>/
|       |-- probe.ckpt
|       |-- probe_config.json
|       `-- metadata.json
`-- dl3dv_geometry/
    `-- <model>/
        |-- probe.ckpt
        |-- probe_config.json
        `-- metadata.json

Each task directory contains one subdirectory per model, such as qwen3-vl-8b or wan-t2v-14b. probe_config.json specifies the probe architecture; metadata.json records the checkpoint's actual feature settings and provenance. manifest.json indexes all checkpoints and their SHA-256 checksums. The semantic-tagging directories additionally include CLIP class-name embeddings and the ordered class vocabulary.

Feature-Layer Clarification

The paper's implementation details specify layer 21 for Qwen2.5-VL and layer 22 for Qwen3-VL in the main 3D experiments. The following uploaded checkpoints were instead trained with layer 18 features:

Model	Layer stated in the paper's main 3D settings	Uploaded checkpoints' feature layer
Qwen2.5-VL-3B	21	18
Qwen2.5-VL-7B	21	18
Qwen3-VL-2B	22	18

For these three models, use layer-18 features to evaluate the uploaded checkpoints across all three tasks. In our spot checks, the layer-18 results match the corresponding numbers reported in the paper.

Citation

@article{shen2026probingvlmvgm,
  title={Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models},
  author={Shen, Haozhan and Zhao, Tiancheng and Zhao, Kangjia and Yin, Jianwei},
  journal={arXiv preprint arXiv:2605.28132},
  year={2026}
}