DOI: 10.5281/zenodo.18682077
title: Sound Source Localization for 3D Spatial Mapping of Surgical Actions in Dynamic Scenes

description: Purpose: Surgical scene understanding is key to advancing computer-aided and intelligent surgical systems. Current approaches predominantly rely on visual data or end-to-end learning, which limits fine-grained contextual modeling. This work aims to enhance surgical scene representations by integrating 3D acoustic information, enabling temporally and spatially aware multimodal understanding of surgical environments.

Methods: We propose a novel framework for generating 4D audio-visual representations of surgical scenes by projecting acoustic localization information from a phased microphone array onto dynamic point clouds from a RGB-D camera. A transformer-based acoustic event detection module identifies relevant temporal segments containing tool–tissue interactions which are spatially localized in the audio-visual scene representation. The system was experimentally evaluated in a realistic operating room setup during simulated surgical procedures performed by experts.

Results: The proposed method successfully localizes surgical acoustic events in 3D space and associates them with visual scene elements. Experimental evaluation demonstrates accurate spatial sound localization and robust fusion of multimodal data, providing a comprehensive, dynamic representation of surgical activity.

Conclusion: This work introduces the first approach for spatial sound localization in dynamic surgical scenes, marking a significant advancement towards multimodal surgical scene representations. By integrating acoustic and visual data, the proposed framework enables richer contextual understanding and provides a foundation for future intelligent and autonomous surgical systems.
description: In this record, we provide the dataset and our baseline sound source locatization predictions.The dataset is contained in ssl_dataset.zip. The dataset contains one file per recording and modality. After extraction, it has the following structure:




audio/
Mono-channel audio in .wav format


audio_event_labels/
Ground-truth surgical action timestamps in .csv format. For non-instantaneous events like sawing and drilling it contains an (N, 2) array of (start, end) timestamps. For chiseling (instantaneous event), it contains an (N, 1) array of frame timestamps.


bbox3d_labels/
Ground-truth oriented 3D bounding boxes of the surgical instrument, provides in .npy (numpy) format. Each file contains an (N, 8, 3) array with the 3D coordinates of the 8 bounding box corners. The bounding boxes were computed based on the object models and the poses obtained via the tracking system.


bbox3dcenter_labels/
Ground-truth oriented 3D bounding boxes centers of the surgical instrument, provides in .npy (numpy) format. Each file contains an (N, 3) array with the 3D center locations of the bounding boxes. This info is derived from the 3D bounding boxes.


object_models/
Reference meshes of the surgical instruments and bones used in the experiment. For the chisel, we additionally provide the contact area bounding box (defined as a cube with 5cm side length, centered around the contact edge)


rgbd/
Stereo RGB videos in .svo2 format (see Stereolabs ZED Documentation: https://www.stereolabs.com/docs/video/recording).


tracking_system/
Marker poses and fiducial positions from the Atracsys FusionTrack 500 tracking system, provided in a custom .csv format.


webcam/
RGB stream from the acoustic camera's integrated webcam.


webcam_acoustic_heatmap/
Acoustic heatmaps projected onto the image plane of the integrated webcam.




 

The baseline predictions are contained in ssl_baseline_predictions.zip. After extraction, it has the following structure:




event_detection/
Detected surgical actions from our event detection module. The format follows the one of the labels in audio_event_labels/.


event_localization2d/
2D bounding box detections based on the acoustic heatmaps, in .csv format. Each detection is provides as (frame_idx, x, y, w, h). Note that these detections were not used nor evaluated in the manuscript, but are provided for completeness.


event_localization3d/
3D bounding boxes from our event localization module, in .npy format. The format follows the one of the labels in bbox3d_labels/.


event_localization3d_baseline/
Ablation study: Predicted sound source locations from the naive localization baseline, in .npy format. The format follows the one of the labels in bbox3dcenter_labels/.