3D CoCa: Contrastive Learners are 3D Captioners
Authors: name: Ting Huang; name: Zeyu Zhang; name: Yemin Wang; name: Hao Tang
Published: 2025-04-13 11:10:47

Abstract: 3D captioning, which aims to describe the content of 3D scenes in natural
language, remains highly challenging due to the inherent sparsity of point
clouds and weak cross-modal alignment in existing methods. To address these
challenges, we propose 3D CoCa, a novel unified framework that seamlessly
combines contrastive vision-language learning with 3D caption generation in a
single architecture. Our approach leverages a frozen CLIP vision-language
backbone to provide rich semantic priors, a spatially-aware 3D scene encoder to
capture geometric context, and a multi-modal decoder to generate descriptive
captions. Unlike prior two-stage methods that rely on explicit object
proposals, 3D CoCa jointly optimizes contrastive and captioning objectives in a
shared feature space, eliminating the need for external detectors or
handcrafted proposals. This joint training paradigm yields stronger spatial
reasoning and richer semantic grounding by aligning 3D and textual
representations. Extensive experiments on the ScanRefer and Nr3D benchmarks
demonstrate that 3D CoCa significantly outperforms current state-of-the-arts by
10.2% and 5.76% in CIDEr at 0.5IoU, respectively. Code will be available at
https://github.com/AIGeeksGroup/3DCoCa.