Title: SurgRe4DGS: Open-Vocabulary Vision-Language Gaussian Splatting for Retrievable 4D Surgical Scene Reconstruction
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 4D Surgical Scene Reconstruction, Surgical Instrument Retrieval, 4D Gaussian Splatting, Vision-language

Abstract: Reconstructing dynamic 4D models from surgical workflows is crucial for robotic-assisted surgery, real-time navigation, and postoperative analysis. However, real-world challenges encountered during surgery, such as tissue deformations and occlusions, significantly increase the difficulty of reconstruction and semantic perception. In this work, we introduce SurgRe4DGS, a novel framework that jointly realizes 4D surgical scene reconstruction and open-vocabulary instrument retrieval through synergistic vision–language Gaussian representations. Particularly, we leverage visual clues to enhance color representations through spatial-color blending with temporal modulation, while language features are instilled with hybrid injection strategies via semantic map rendering and weighted top-K aggregation with contrastive learning. With a dual-branch mechanism for vision-language context integration, SurgRe4DGS ensures precise cross-modal alignment and semantic consistency in deformable scenarios. To fill the gap in existing benchmarks for language-driven multimodal 4D reconstruction and open-vocabulary instrument retrieval, we introduce StereoLung15K, a pioneering surgical dataset with 26 binocular thoracoscopic sequences, over 15,000 annotated frames, and multimodal (RGB/depth/text) support at 30 FPS, enabling joint reconstruction and text-guided retrieval tasks. Extensive experiments on two public datasets, StereoMIS and EndoNerf, as well as our StereoLung15K, demonstrate that SurgRe4DGS achieves state-of-the-art performance in terms of reconstruction fidelity and retrieval precision. 
Our code and data will be publicly available.