DOI: 10.5061/dryad.3ffbg79zd
title: Data and code from: Deep learning-based autonomous retinal vein cannulation in ex vivo porcine eyes

description: Retinal Vein Cannulation (RVC) is an emerging method for treating Retinal
 Vein Occlusion (RVO). The success of this procedure depends on surgeon
 expertise and, recently, robotic assistance. This paper proposes an
 autonomous RVC workflow leveraging deep learning and computer vision. Two
 Steady Hand Eye Robots (SHERs) control a 100-micrometer metal needle and a
 medical spatula to execute precise tasks. Three convolutional neural
 networks are trained to predict needle movement direction and identify
 contact and puncture events. A surgical microscope with an intraoperative
 Optical Coherence Tomography (iOCT) system captures the surgical field
 through a microscope and cross-sectional images. The goal is to enable the
 robot to autonomously carry out the critical steps of the RVC procedure,
 especially those that are challenging and require expert knowledge. The
 less technically demanding tasks are assigned to the user, who also
 supervises the robot during these steps. Our method is tested on 20 ex
 vivo porcine eyes, achieving a success rate of 90 %. Additionally, we
 simulate eye movements caused by breathing on six other ex vivo porcine
 eyes. With the eyes moving in a sinusoidal pattern, we achieve a success
 rate of 83 %, demonstrating the robustness and stability of the proposed
 workflow. Our results demonstrate that the autonomous RVC workflow,
 incorporating deep learning and robotic assistance, achieves high success
 rates in both static and dynamic conditions, indicating its potential to
 enhance the precision and reliability of RVO treatment.
description: # Deep Learning-Based Autonomous Retinal Vein Cannulation in ex vivo
 Porcine Eyes # Introduction This repository contains all necessary code
 and data for reproducing the training results and control algorithms for
 the paper titled "Deep Learning-Based Autonomous Retinal Vein
 Cannulation in ex vivo Porcine Eyes". Please
 unzip contact_network.zip, dataset_for_all.zip, navigation_network.zip, puncture_network.zip, and SHER_visualiaztion_and_control.zip first. # Installation We recommend running this in a virtual environment: ``` # generate a virtual environment with name test_env and Python 3.8.16 installed conda create -n test_env python=3.8.16 # activate the environment conda activate test_env # deactivate the environment conda deactivate # delete the virtual environment and all its packages conda remove -n test_env --all ``` To install all necessary packages, please navigate to the cloned directory and run the following code in the terminal: ``` pip install -r requirements.txt ``` # Navigation Network Train the navigation network using navigation_network_train.ipynb. Validate and test accuracy using navigation_network_val_test.ipynb. # Contact Network Train, validate, and test the contact network using contact_network_train_val_test.ipynb. # Puncture Network Train the puncture network using puncture_network_train.ipynb. Validate and test accuracy using puncture_network_val_test.ipynb. # SHER Visualiaztion and Control **The code in this folder was developed for Steady Hand Eye Robot (SHER) at Johns Hopkins University only.** If you need to use the code for your own robot system, please adjust the code accordingly. Following the below steps in the terminal to achieve autonomous retinal vein cannulation with fixed eyes and vertical sinusoidal motion and compensation. * subscribe microscope image through HD SDI output and publish to /decklink/camera/image_raw ros topic using gscam package. * run the keyboard contorller code: ``` sudo -s cd ./keyboard_controller/ source devel/setup.bash # publish the key commonds rosrun key_publisher key_publisher.py ``` * open a new terminal: ``` cd ./keyboard_controller/ source devel/setup.bash rosrun key_move key_move.py ``` ## For Autonomous RVC with Fixed Eyes ``` # publish B-scans python3 visualization_b_scans.py # visualization of the network output on microscope image and B-scan python3 visualization.py # control and inference python3 visualization_inference.py # data saving python subscribe_all.py ``` ## Autonomous RVC with Vertical Sinusoidal Motion and Compensation ``` # publish B-scans python3 publish_b_scans_for_breath_simulation.py # visualization of the network output on microscope image and B-scan python3 visualization.py # initialize the XYZ linear stage python xyz_stage_move_to_position.py # simulate the breathing using the XYZ linear stage python xyz_stage_rvc_z_axis_sinewave_motion_recording.py # control and inference python3 breath_simulation_inference.py # data saving python subscribe_all.py ``` ## For Breathing Simulation Only - Show How It Works ``` python3 breath_simulation.py ```