Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial
  Representations
Authors: name: Yujia Zhang, name: Xiaoyang Wu, name: Yixing Lao, name: Chengyao Wang, name: Zhuotao Tian, name: Naiyan Wang, name: Hengshuang Zhao
Organization: Pointcept

Humans learn abstract concepts through multisensory synergy, and once formed,
such representations can often be recalled from a single modality. Inspired by
this principle, we introduce Concerto, a minimalist simulation of human concept
learning for spatial cognition, combining 3D intra-modal self-distillation with
2D-3D cross-modal joint embedding. Despite its simplicity, Concerto learns more
coherent and informative spatial features, as demonstrated by zero-shot
visualizations. It outperforms both standalone SOTA 2D and 3D self-supervised
models by 14.2% and 4.8%, respectively, as well as their feature concatenation,
in linear probing for 3D scene perception. With full fine-tuning, Concerto sets
new SOTA results across multiple scene understanding benchmarks (e.g., 80.7%
mIoU on ScanNet). We further present a variant of Concerto tailored for
video-lifted point cloud spatial understanding, and a translator that linearly
projects Concerto representations into CLIP's language space, enabling
open-world perception. These results highlight that Concerto emerges spatial
representations with superior fine-grained geometric and semantic consistency.