Geometrically constrained sub-pixel disparity estimation from stereo images of the retinal fundus

The aim of this study is to propose a precise disparity estimation process between retinal fundus stereoscopic images to assist during eye diagnosis and view generation for visualization. We propose a computer-vision-based solution that allows, from stereo images, the extraction of precise disparity information that could be used for clinical parameters estimation and/or the generation of multi-viewpoint images of retinal fundus. The proposed solution is based on a sub-pixel disparity estimation algorithm that takes into consideration the spherical shape of the retinal fundus. A sub-pixel approach is adopted in order to achieve high precision disparity estimation. Moreover, the a priori knowledge of the fundus geometric shape provides useful information for the regularization of the high precision disparity estimation process. Stereo images, with known ground truth, are used to evaluate the proposed algorithm and to demonstrate the high precision of estimated disparity in our methods. Introduction 3D retinal surface visualization is becoming easy and handy with the current advances in image visualization technologies. Retinal image processing is a well-studied field and several studies about depth estimation of retinal fundus exists [1, 2, 3, 4]. The literature for the analysis of retinal images mainly deals with the issue of 2D images in order to facilitate the detection of lesions by ophthalmologists. However, accurate diagnosis often requires a 3D image of the retina. In particular, the analysis of the 3D shape of retinal fundus is essential for identifying lesions and estimating the extent of the lesion, and measuring eye pressure on the optic disc. Different methods are used to regenerate a 3D model of the fundus from stereo images. In [3], the authors estimate the epipolar geometry and projection matrices after a self-calibration, then they solve the correspondence problem to reconstruct the 3D fundus surface. In [7], the authors apply a plane+parallax algorithm to stereo images, which is followed by a mutual information-based disparity search stage. In [8], the authors use a multi-focusing technique to capture retinal fundus images and reconstruct its 3D surface. None of these studies assume that the retinal fundus can be approximated to a spherical surface. In [5, 6] the authors assume a spherical shape for the eyeball. They use this information to help reconstruct the 3D shape of the fundus. One challenging step is to estimate the intermediate optical system parameters such as the contact enlarging lens focal distance. They model the mapping of this surface through the eye lens as a quadratic surface and perform camera calibration through matching correspondences on this quadratic surface. Their method is based on a two-stage optimization process. They estimate the quadratic surface using the correspondences and they minimize the error with respect to the camera poses by keeping the surface equation fixed. Secondly, they minimize the error with respect to the surface equation and camera poses to estimate the optical system parameters. This work aims to reconstruct the 3D shape of the retinal fundus, and does not use the estimated quadratic surface to increase the robustness of the disparity estimation process. On the other hand, the authors in [10], state that the extraction of invariant geometric features is the first step not only for the 2D registration but also for the 3D reconstruction of retinal images. They proposed a robust method, using reliable landmarks, defined by the Y-feature , where vessels and nerves intersect. They used this method to extract robust corresponding points in order to rectify uncalibrated stereo images and then estimate the disparity without taking into consideration the quadratic surface constraint. However, for establishing disparities, the image correspondence problem must be solved throughout feature matching. Matching techniques, and thus constructing the disparity map, can be classified into two main categories. More specifically, techniques that construct the disparity map by solving the correspondence problem, pixel by pixel, are referred to as local, while techniques that consider the correspondence problem as a global optimization problem are referred to as global. Among the best-known techniques belonging into the mentioned categories are the differential matching, the cross correlation, graph cuts, the global energy optimization method and dynamic programming based methods to name a few. All local methods typically use an appropriate measure in order to quantify the existing similarity between the template window and the candidate one. Widely used similarity measures are the sum of squared differences (SSD), the sum of absolute differences (SAD), and the normalized cross-correlation (NCC) as well as their zero-mean counterparts. Among these measures, only the zero-mean normalized cross correlation is invariant to both shift and scale photometric distortion. This property is required in many stereo vision algorithms, especially in ophthalmic applications where the illumination of the scene is nonuniform. Another feature which is desired in a large number of applications is the ability of the matching algorithm for producing a disparity map with sub-pixel accuracy. Sub-pixel accuracy allows for a finer variation in depth levels, which otherwise are limited to values corresponding to integer disparities. This improves the precision of the depth estimation process and thus the quality of the synthesized images for multi-viewpoint visualization. In [9] the authors propose a new similarity measure which is based on the correlation coefficient called Enhanced Normalized ©2016 Society for Imaging Science and Technology DOI: 10.2352/ISSN.2470-1173.2016. 5.SDA-034 IS&T International Symposium on Electronic Imaging 2016 Stereoscopic Displays and Applications XXVII SDA-034.1 Cross Correlation (ENCC). More specifically, by using an appropriate linear interpolation scheme on the intensities of two adjacent candidate windows, a new similarity measure is introduced. This measure, although based on a linear interpolation scheme, does not demand the reconstruction of any intensity value, while at the same time it has infinite precision in its sub-pixel estimates. In this paper, we propose a new algorithm for precise and robust sub-pixel disparity estimation exploiting the a priori knowledge of the retinal fundus shape. This method consists of, first, extracting reliable landmarks, defined by Y-feature points in stereo images. These points are then matched and used to generate a quadratic model of the retinal fundus. Secondly, a global subpixel disparity estimation process is developed under a geometric constraint defined by the quadratic surface. The rest of the paper is organized as follows: Section 2 introduces the geometric model of the fundus surface and the its parameters estimation process. Section 3 describes the matching algorithm for sub-pixel disparity map. Section 4 discusses the algorithm implementation and its results. The last section concludes the paper and outlines future research directions. Geometric constraint for fundus stereo images In this paper we will consider the stereoscopic setup presented in figure 1. The two cameras are modeled as Pin-Hole in a canonical stereoscopic system. The stereo images are considered rectified, thus any point in the space will be mapped to corresponding image points through fundamental matrices Ml and Mr at the same vertical position. Disparity geometric model Assuming an orthogonal frame O,− →x ,− →y ,− →z and a retinal fundus (modeled as a sphere) with radius R, the coordinates (X ,Y,Z) of a point M lying on the fundus surface should verify the sphere equation: (X−Cx) +(Y −Cy) +(Z−Cz) = R2, (1) where Cx, Cy and Cz are the coordinates of the sphere-shaped fundus center. The figure 2 shows the spherical shape and the optical path of a retinal fundus stereoscopic image capturing system. In [5] the author demonstrates that the point M is mapped through the optical systems into a point p which coordinates verify a quadratic equation of the form: x2 + y2 +αzz +αxzxz+αyzyz+αxx+αyy+αzz+α = 0. (2) The projection of p to both camera plans Πl and Πr is computed using the cameras fundamental matrices Ml and Mr. This will give two image points ml(ul ,v) = Ml × pt and mr(ur,v) = Mr× pt in the left and right image coordinate system of the left and right cameras respectively. The disparity between ml and mr is defined by d = ul −ur. We can demonstrate that: