# CT-Video Matching for Retrograde Intrarenal Surgery Based on Depth Prediction and Style Transfer
## 2 Materials and Methods  
Due to the limitations of traditional methods depicted above, a new method of preoperative CT and intraoperative video information fusion was proposed to solve the CT-video matching problem in ureteroscopic surgical guidance in this paper. Inspired by the application of deep learning and depth maps in the field of endoscopic guidance [8; 9], we proposed to use depth maps as the intermediate connecting medium to match preoperative CT with intraoperative video. Our proposed method could be specified through the following steps.  
1. We first reconstructed a 3D model of the kidney based on CT images.
2. We used a virtual camera to simulate the movement path of the real ureteroscope to generate pairs of images. Each image pair consists of one simulated image (SI) taken by a virtual camera and its corresponding depth map (DM). We call these pairwise images datasets (simulated images and depth maps) as SI-DM. Based on the SI-DM dataset, we trained a model to predict the depth of simulated images.
3. We trained a model to transfer the style of endoscopic images (EI) into the style of simulated images mentioned in (2). Then we could indirectly obtain endoscopic images' depth maps.
4. Finally, by extracting features of the depth maps from SI and EI and calculating their similarity, we realized CT-video matching to avoid kidney stone misdiagnosis based on the depth maps from (2) and the depth maps from (3).  
In this method, both the depth prediction network and the style transfer network are crucial. The style transfer network can transfer one modal data to another modal data, which means that if we know one modal data's depth maps, another modal data's depth maps can be predicted indirectly. This approach has been shown to be effective when applied to natural images [10]. Moreover, to avoid using traditional feature extraction methods to compute feature descriptors for matching, we use convolutional neural networks (CNNs) instead to extract deep semantic features of the depth map, which can improve the time performance.  
The main contributions of our paper are as follows:  
* We established a corresponding mapping relationship based on the depth map between the white-light ureteroscopic image and the virtual endoscopic image. In other words, we achieved depth prediction based on a single white-light endoscopic image.
* We extract abstract semantic features of the depth map from ureteroscopic images and simulated images captured by the virtual camera for CT-video matching. This approach achieves effective matching and significantly reduces the computational time consumption.
* The results show that our method achieved a 26% improvement in top 10 matching accuracy with a five times improvement in time performance.  
Figure 1 shows the flow chart of our approach to navigate the ureteroscope by matching preoperative CT images and intraoperative video frames. In detail, first, a 3D model of the kidney was reconstructed by CT images. Then we generated an SI-DM dataset based on the 3D kidney and virtual reality to train the depth prediction model. Meanwhile, we introduced a style transfer network to transform ureteroscopic images into simulated images, whose style was like the SI dataset. Once we obtained both ureteroscopic images' depth maps and SI dataset's depth maps, we could extract their depth semantic features, respectively. Finally, based on the depth maps' features, we realized a CT-video matching task to make sure each branch of the kidney was examined.