Title: Segmentation Helps Understanding: Mask-Infused Vision-Language Pre-training for 3D Medical Images
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: 3D medical image pretraining, contrastive learning

Abstract: Pretraining effective 3D medical image encoders is essential for advancing downstream tasks such as diagnosis and prognosis. Recent contrastive language-image pretraining approaches have leveraged paired radiology reports to learn high-level semantic features from 3D scans. However, these methods often overlook fine-grained, clinically significant details, such as small lesions. In this work, we explore the underutilized potential of segmentation data, which, despite being unlinked to radiology reports, naturally contain precise spatial information. We propose a unified contrastive learning framework, SegVL, that infuse segmentation masks into vision-language pre-training to obtain a single, robust 3D image encoder. Our method introduces voxel-mask contrastive learning using segmentation mask names as text anchors to refine voxel embeddings, and enhances image-text contrastive learning through segmentation-informed image representations. We also employ a Tversky loss to mitigate extreme class imbalance and highlight the importance of a lightweight decoder to preserve encoder capacity. Extensive experiments across multiple external datasets and tasks demonstrate that SegVL consistently outperforms existing pretraining strategies, validating the synergy of segmentation and VL data in learning comprehensive 3D medical representations.