# Fully Leveraging Deep Learning Methods for Constructing Retinal Fundus Photomatages  
[PERSON]  
1 Department of Ophthalmology, Seoul National University College of Medicine, Seoul National University Bundang Hospital, Seongming 13620, Korea; [EMAIL_ADDRESS] (J.K.); [EMAIL_ADDRESS] (S.G.); [EMAIL_ADDRESS] (K.J.N.). [PERSON]  
1 Department of Ophthalmology, Seoul National University College of Medicine, Seoul National University Bundang Hospital, Seongming 13620, Korea; [EMAIL_ADDRESS] (J.K.); [EMAIL_ADDRESS] (S.G.); [EMAIL_ADDRESS] (K.J.N.). [PERSON]  
1 Department of Ophthalmology, Seoul National University College of Medicine, Seoul National University Bundang Hospital, Seongming 13620, Korea; [EMAIL_ADDRESS] (J.K.); [EMAIL_ADDRESS] (S.G.); [EMAIL_ADDRESS] (K.J.N.). [PERSON]  
1 Department of Ophthalmology, Seoul National University College of Medicine, Seoul National University Bundang Hospital, Seongming 13620, Korea; [EMAIL_ADDRESS] (J.K.); [EMAIL_ADDRESS] (S.G.); [EMAIL_ADDRESS] (K.J.N.). [PERSON]  
2 School of Electrical Engineering, Kookmin University, Seoul 02707, Korea 22 School of Electrical Engineering, Kookmin University, Seoul  
The photomontage technique does exactly this by aligning multiple images through image registration, and then stitching and blending them into a single image [7; 8; 9; 10; 11]. In the work of [PERSON] et al. [7], without many techniques for registration, background subtracted images are warped using a polynomial function that is based on guided matches. In [PERSON] et al. [8], a fully automated method that is based on a hierarchical matching scheme with a model of weak perspective camera, rigid motion, and approximate quadratic surface of the retina was proposed. In [PERSON] et al. [9], the improved SURF keypoint matching method [12] was used to improve the matching and, thus, the montage. In [PERSON] et al. [10], the primary focus is on the modeling and validating the geometry of the eye. In a more recent work by [PERSON] et al. [11], deep learning is indeed used as a CNN, but only to segment the vessel map from which bifurcation features are localized and matched for registration. There are several issues that still can be improved within the frame of integrating retinal fundus images to achieve a wider field of view. One is the order and combination of the image alignment. As the number of images to be integrated increase, the combinatorial space of optimally aligning the images may become very large. As most image registration methods, such as feature point based methods [13], operate on pairs of images, it is simple to sequentially register the next image to the montage one-by-one. Here, determining the optimal order of images to be subsequently added arises as one key problem. Another issue is the accuracy of image registration and image warping during alignment and stitching. As the images are from the retina, distortion from projecting the three-dimensional spherical shape into two-dimensions increase as the stitched images are positioned farther from the center of the montage. The image registration and warping must be as accurate as possible in order to minimize artifacts. In this paper, we present a novel framework for constructing photomontages of the retina from multiple fundus images that fully leverages recent deep learning methods. We apply a deep learning based object detection method [14] to detect the optic disc and retinal fovea, which is then used as reference landmarks to determine the order of image registration and alignment. We also adopt a recently proposed two step image registration framework [15; 16] comprising rigid and non-rigid registration, but with key modifications to improve robustness for images in the peripheral of the retina. Here, CNN based vessel segmentation is used as the basis to maximize the accuracy of both rigid and non-rigid registration. Figure 1 depicts the overview of the proposed framework comprising the landmark detection, order determination, and iterative registration.