DOI: 10.5061/dryad.8gtht76x9
title: Dataset and trained models for video denoising in fluorescence guided surgery

description: Fluorescence guided surgery (FGS) is a promising surgical
 technique that gives surgeons a unique view of tissue that is
 used to guide their practice by delineating tissue types
 and diseased areas. As new fluorescent contrast agents are
 developed that have low fluorescent photon yields, it becomes
 increasingly important to develop computational models to allow
 FGS systems to maintain good video quality in real time
 environments. To further complicate this task, FGS has a
 difficult bias noise term from laser leakage light (LLL) that
 represents unfiltered excitation light that can be on the order
 of the fluorescent signal. This dataset contains the data used to
 develop and train video denoising models for fluroescence guided surgery
 with LLL. This dataset contains bright fluorescence data in a mock chicken
 thigh surgery for FGS video simulation, non-fluorescent video for LLL
 simulation, as well as a number of calibration datasets for properly
 simulating a comercial system, and real noise video for testing. We also
 provide result videos of our denoising models trained with this data and
 the trained models.
description: This dataset contains fluorescence and reference (RGB) video of a
 mock chicken thigh surgery captured on the OnLume Avata commercial
 fluroescence guided surgery (FGS) system using indocyanine green (ICG) as
 the contrast agent. It contains a number of different conditions used to
 calibrate a realistic noise model and simulate real noisy data for the
 training of deep learning based video denoising models. 
 First, this dataset contains a number of videos
 captured with high concentration of ICG that can act as a basis for
 simualtion and as a ground truth in training. This portion of the dataset
 is an expansion of the dataset in [1], and contains the videos from [1]
 along with another ~100 minutes of video. Second, our
 dataset contains 15 minutes of real noisy data of a mock chicken thigh
 surgery with low concentration of ICG. Third, our
 dataset contains 20 minutes of mock chicken thigh surgery with no ICG that
 can be used to simualte laser leakage light. Finally,
 our dataset contains a number of videos used to calibrate our noise model
 to properly simualte the OnLume Avata system noise. 
 [1] Seets, Trevor et al. (2024). Data for OFDVDnet: A sensor
 fusion approach for video denoising in fluorescence guided surgery
 [Dataset]. Dryad. https://doi.org/10.5061/dryad.v6wwpzh3w
description: # Dataset and trained models for Video Denoising in Fluorescence Guided
 Surgery
 [https://doi.org/10.5061/dryad.8gtht76x9](https://doi.org/10.5061/dryad.8gtht76x9) This repository contains the data necessary for calibrating an accurate noise model for the OnLume Avata system, a fluorescence guided surgery commercial system. It also contains a number of low-noise fluorescent video of a mock chicken thigh surgery that can be used to simulate realistic noisy videos which can be used to train video denoising models for fluorescence guided surgery.  ## Description of the data and file structure There are three main folders in our dataset: /data : this folder contains our dataset videos. Each video is in "triple view" mode where the bottom left quadrant is the reference video, the top right is the fluorescence video and the top left is the fluorescence video overlayed in false color on the reference video. The full videos are 2048x1536 at 15 fps, each quadrant is 1024x768. Videos can be viewed with VLC video player (https://www.videolan.org/) or most other suitable video player. /data has 4 subfolders 1. /data/lowdose_chicken : contains 2 videos of a mock chicken thigh surgery with a low concentration of ICG injected. These videos make up our OL-Real dataset and have real system noise. 2. /data/no-icg : contains 4 videos split into 3 training videos and 1 testing video make up the OL-LLL dataset. These videos are of a mock chicken  thigh surgery with no icg injected, so the fluorescence video contains only laser leakage light.  3. /data/quel_phantom : contains 2 videos that make up the OL-Phantom. These videos contain videos of the Quel phantom (RCS-ICG-ST01-QUEL0) which is used to calibrate our noise model. 4. /data/train_test : contains 3 subfolders, our pretrained LLL-PN .pt files, and our train test split file. 1. train_test.pt : this file represents a pytorch dictionary and is loaded in our training code. This file contains the indexes used to split the training/testing videos into 100 frame chunks in the network training loops. It is required to run the training and testing code in the referenced zenodo code base. 2. NAF_32_Standard_LLv2__h256_w256_laser_leak_ep100_L1_loss.pt : this is our pre-trained laser leakage light prediction network (LLL-PN) used to simulate laser leakage light in our simulation pipeline. NAF_32_Standard_LLv2__h256_w256_laser_leak_ep100_L1_loss_reverse_test_train.pt is a pre-trained LLL-PN with different testing and training sets and is used to test the robustness of our models to changing LLL. These models are loaded by our simulation models in the zenodo code base, the code required to run and load these models is in the zenodo code base. 3. /data/train_test/dark_frames : contains the OL-Dark dataset. This folder contains a number of dark frame videos that capture the read noise of our system that is used in our simulation method. This folder contains 2 unique videos which are each duplicated 3 times for training  (train.mp4, train1.mp4 and OL-2023-09-15-143614-000002-record.mp4) and testing (test.mp4, test1.mp4, and OL-2023-09-15-142907-000001-record.mp4). These are duplicated to allow parallel training or testing of multiple models without creating read conflicts. 4.  /data/train_test/test : contains the triple view videos used in our test set 5. /data/train_test/train: contains the triple view videos used in our training set /models : this folder contains 5 pretrained models. BL_AM, BL_RNN, and BL_SW are our baseline models we created for this problem. NAF_32_Standard_h256_w256_OL24_ep3000_lowlr_1600.pt is a partially trained NafNet32 model used in the pretraining step of our BL_AM model. NAF_32_Standard_h256_w256_OL24_ep3000_lowlr_final.pt is the final NafNet32 model that can be used as a comparison to our baseline models. These models are pytorch .pt files and the python model code is in the referenced zenodo code base as well as examples on how to load these models into pytorch.  /results : this folder contains 4 result videos at different simulated noise levels. It also contains metric_results.pkl which contains 5 different model metrics on simulated data per test set video. This pkl file is a pandas dataframe and can be opened in pandas, the attached zenodo code contains and example jupyter notebook on how to read and process this pkl file as well as the code to generate the result videos and pkl file.  ## Sharing/Access information Part of this dataset is derived from Seets, Trevor et al. (2024). Data for OFDVDnet: A sensor fusion approach for video denoising in fluorescence guided surgery [Dataset]. Dryad. [https://doi.org/10.5061/dryad.v6wwpzh3w](https://doi.org/10.5061/dryad.v6wwpzh3w) ## Code/Software The provided python code can be used to use this data to simulate noisy data and train our denoising models. The README provided with the code contains information on running the code.