Automatic optical‐to‐SAR image registration using a structural descriptor

IET Image ProcessingVolume 14, Issue 1 p. 62-73 Research ArticleFree Access Automatic optical-to-SAR image registration using a structural descriptor Sourabh Paul, Corresponding Author Sourabh Paul sourabhpaul26@gmail.com orcid.org/0000-0003-2421-0232 Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela, IndiaSearch for more papers by this authorUmesh C. Pati, Umesh C. Pati Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela, IndiaSearch for more papers by this author Sourabh Paul, Corresponding Author Sourabh Paul sourabhpaul26@gmail.com orcid.org/0000-0003-2421-0232 Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela, IndiaSearch for more papers by this authorUmesh C. Pati, Umesh C. Pati Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela, IndiaSearch for more papers by this author First published: 01 January 2020 https://doi.org/10.1049/iet-ipr.2019.0389Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Optical-to-synthetic aperture radar (SAR) image registration is a challenging task in remote sensing as the images have significant non-linear intensity variations as well as large geometric differences. Moreover, the influence of speckle noise in the SAR image further affects the registration result. The structural descriptors are very effective to handle the non-linear intensity variations between optical and SAR images. Although a number of optical-to-SAR image registration methods have been proposed in the past few years based on the structural information of image, most of them are ineffective for the images having large geometric differences. To address these problems, a novel optical-to-SAR image registration algorithm is proposed by using a new structural descriptor. Initially, the corner features are extracted from the optical and SAR images. Then, the proposed structural descriptor is constructed for the extracted features. Finally, feature matching is performed between the optical and SAR images and the correct match are identified. The proposed method is very effective to register the optical and SAR images having significant intensity variations and large geometric differences. It can increase the number of correct match between the images. Experiments on six sets of optical and SAR image pairs demonstrate the effectiveness of the proposed method. 1 Introduction Optical-to-synthetic aperture radar (SAR) image registration performs the alignment of optical and SAR images of a scene taken at a different time from different viewpoints. It has a wide range of remote sensing applications such as image fusion, change detection, and image mosaicing. The optical images are captured by the passive optical sensors, whereas the SAR images are captured by the active SAR sensors. As the imaging principles of the two types of sensors are completely different, their images have different appearances. Moreover, the images captured by the two types of sensors generally have significant geometrical differences. Also, the SAR images contain multiplicative speckle noise which also affects the performance of a registration method. Owing to the factors above, optical-to-SAR image registration is considered as a challenging task. Generally, optical-to-SAR image registration methods are divided into two categories: intensity-based methods and feature-based methods. Intensity-based methods estimate a transformation model which provides the maximum similarity measurement value between the optical and SAR images. Normalised cross-correlation [1] and mutual information (MI) [2] have been used popularly as similarity matrices in the intensity-based methods. However, the main disadvantage of these methods is the high computational complexity. Moreover, these are ineffective for the images having large geometric differences. In the feature-based methods, some robust features such as points, lines, and contours are extracted from the input images, and then a transformation model is estimated by using the correct feature correspondences. These methods are comparatively faster and more effective to handle the large geometric differences between the images than the intensity-based methods. Scale-invariant feature transform (SIFT) [3] has been extensity used as a feature extraction method for the registration of remote sensing images. However, repeatability of the standard SIFT features is not satisfactory enough for optical-to-SAR image registration [4]. Moreover, the standard SIFT algorithm extracts unevenly distributed features. In [5], a uniform robust SIFT (UR-SIFT) algorithm is proposed to uniformly distribute the SIFT features. Fan et al. [4] proposed an improved SIFT (I-SIFT) method to enhance the performance of the SIFT algorithm. In [4], SIFT features are extracted starting from the second octave to improve the repeatability rate of the features. In addition, multiple supported regions are used to construct a descriptor which is more distinctive than the standard SIFT descriptor. In [6], a modified SIFT algorithm is presented to register the remote sensing data sets which include optical and SAR images. In [7, 8], the Harris detector is used to extract the corner features for optical-to-SAR image registration. Multi-scale Harris operator [9] has been widely used to extract corner features from the remote sensing images. In [10], Fan et al. proposed a uniform non-linear diffusion-based feature extraction method by using the multi-scale Harris operator. Li et al. [11] proposed an optical-to-SAR image registration method by using the contour feature of image. In [12], Sui et al. extracted the line segments from the optical images for optical-to-SAR image registration. In this method, the extracted line segments are matched by using Voronoi diagram and multi-resolution analysis. However, it is applicable for the image pairs which can provide the linear features. Xiang et al. [13] presented an optical-SAR-SIFT algorithm to register the high-resolution images. In [13], the gradients in the optical image and the SAR images are computed by the Sobel operator and ratio of exponentially weighted average [14] operator, respectively, to improve the image matching result. Chen et al. [15] proposed an optical-to-SAR image registration method by using the phase congruency (PC) [16] and Gaussian gamma-shaped bi-directional window (GGSBW)-based [17] descriptor to register the optical and SAR images. In this method, the repeatability of the extracted features is increased by PC analysis and the distinctiveness of the descriptor is improved by GGSBW-based gradient computation. A variety of methods have been proposed in the past decades to improve the distinctiveness of the feature descriptor for multi-modal image registration. In [18], Mikolajczyk and Schmid presented a gradient location and orientation histogram descriptor which gives better matching results than the standard SIFT descriptor. Chen et al. [19] proposed a partial intensity invariant feature descriptor which is constructed by considering the gradient direction from 0 to . In [20], the SIFT descriptor is constructed by limiting the gradient direction from 0 to . Speed-up robust feature (SURF) [21] is a well known mono-modal image matching approach. However, it is ineffective for multi-modal images. Zhao et al. [22] proposed a multi-modal-SURF descriptor for multi-modal image registration. However, these methods are a gradient-based method which is less effective to handle non-linear intensity variations between optical and SAR images [10]. In [7, 8, 10], different structural descriptors are proposed for multi-modal image registration. These descriptors are based on local self-similarity (LSS) [23] and very effective to handle the non-linear intensity differences between the images. The LSS utilises the local shape property of the image to construct the descriptor which can handle the non-linear intensity differences between images. Fan et al. [10] presented an optical-to-SAR registration algorithm by using corner features and PC-based structural descriptor. In [10], initially, homogeneously distributed corner features are extracted from the input images and structural descriptors are formed for the extracted features. Then, feature matching is performed to identify the correct match. Ye et al. [7] proposed an optical-to-SAR image registration method using block-based Harris features [24] and an improved LSS descriptor, known as dense LSS. Ye et al. [8] proposed a registration algorithm based on Harris features extraction and structural similarity. In [25], a distinctive order-based SS (DOBSS) descriptor is presented to register the remote sensing optical images having significant geometric differences. In this method, feature extraction is performed by the UR-SIFT algorithm. However, the repeatability of the SIFT features is very less in optical-to-SAR image registration [8]. Although a number of structural descriptors have been proposed in recent years for optical-to-SAR image registration, most of them are not applicable for the image pairs having significant geometric differences. To address this problem, an optical-to-SAR image registration method is proposed in this paper. To improve the repeatability rate of the extracted features, a multi-scale Harris function is used along with a Gabor odd filter-based ratio operator (GOFRO) to extract corner features from the optical and SAR images. Then, one or more dominant orientations are assigned to every feature to achieve rotation invariance. After that, a structural descriptor is constructed for each of the features by using PC information. To reduce the effect of noise in the structural descriptor, a pre-filtering operation is performed by using an adapted anisotropic Gaussian (AAG) filter. In addition, a novel approach is presented to improve the distinctiveness of the proposed descriptor. The main contributions of this paper are listed as follows: (i) An automatic optical-to-SAR image registration algorithm is proposed to register the optical and SAR images having significant geometric as well as non-linear intensity differences. (ii) The GOFRO is used along with a multi-scale Harris function to extract corner features from the input images. The corner features are extracted to improve the repeatability rate of the extracted features in optical-to-SAR image registration. (iii) A structure descriptor is constructed to handle significant intensity differences and the noise effect. The advantages of the proposed method are as follows: (i) It is robust to significant geometric as well as intensity differences between the optical and SAR images. (ii) It can increase the number of correct match (NCM) in optical-to-SAR image registration. (iii) It can improve the distinctiveness of the descriptor (1) Rest of this paper is organised as follows: In Section 2, the proposed optical-to-SAR image registration method is discussed. Section 3 provides the experimental results of the proposed method. Finally, this paper is concluded in Section 4. 2 Proposed method Fig. 1 shows the framework of the proposed optical-to-SAR image registration method. At first, the GOFRO is used along with multi-scale Harris function to extract corner features from the optical and SAR images. Then, the input images are filtered by an AAG filter to reduce the effect of noise. The dominant orientations of the extracted features are determined by considering a local area around every feature. The local area is selected from the filtered image. After that, a structural descriptor is constructed for each of the features. Then, feature matching is performed between the optical and SAR images. The correct match obtained by the matching process are used to estimate the transformation between input images. The SAR image is transformed by using the estimated transformation. Details of these steps are provided in the next sections. Fig. 1Open in figure viewerPowerPoint Framework of the proposed method 2.1 Corner feature extraction The multi-scale Harris function and its response are mathematically expressed by (1). Here, *, tr, and det denote convolution, trace, and determinant, respectively; G is a Gaussian function having the standard deviation of , where is the current scale. and are the gradients in m and n directions, respectively. The ratio-based operator is generally used to compute the gradients. In [26, 27], it is mentioned that the GOFRO is comparatively more robust than the other ratio operators in the presence of speckle noise. So, in our proposed method, it has been used to compute the gradients for the multi-scale Harris function. This multi-scale Harris function is used to extract corner features from the input images. In a previous research, we have proposed a modified UR-SIFT (M UR-SIFT) algorithm [28] to improve the distribution quality of the extracted features. So, it has been utilised to properly distribute the extracted corner features in our proposed method. 2.2 Filtering by AAG filter After feature extraction, one or more dominant orientations are assigned to every feature point. To estimate the dominant orientation and to construct the descriptor, a circular local supported region is considered around every feature point. The radius of the local supported region is determined by the M UR-SIFT algorithm. In many methods [23, 25], the local supported region is directly selected from the original image. However, it affects the performance of the structural descriptor as the remote sensing images contain noise. Therefore, in our proposed method, the local supported region is selected from the filtered image to reduce the effect of noise. Filtering is performed by an anisotropic filter which preserves more details than the standard Gaussian filter. Wang et al. [29] proposed an AAG filter to improve the performance of SIFT operator. The AAG filter is comparatively more effective to remove the noise in images than the non-linear diffusion-based anisotropic filter [29]. Therefore, it has been used in our proposed method. An oriented AAG filter is shown in Fig. 2. The AAG filter uses the elliptical Gaussian mask to smooth the edges and the radial mask is used to smooth the homogeneous regions [29, 30]. It is defined as [31] (2)where the convolution operator is denoted by *; denotes the direction of the AAG filter; and are the scaling parameters in the two orthogonal directions U and Z, respectively; U represents the detection of ; and (3)Here, the value of is set to 2. A very large value of increases the blurring effect. An orientation histogram is constructed by using the sample points around () to estimate the orientation . Here, is determined as orthogonal to the orientation of the maximum peak of the orientation histogram. A second-moment matrix is used to determine the value of . Fig. 2Open in figure viewerPowerPoint Anisotropic Gaussian filter It is defined as (4)Here, and are the gradients at a point () in the m and n directions, respectively. denotes the Gaussian weight function with the scale coefficient . In our proposed method, the GOFRO is used to compute the gradients. The statistical characteristics of a local region is described by a factor known as isotropy () which is calculated as (5)where denotes the minimum eigenvalue of the matrix and denotes the maximum eigenvalue of the matrix . These values are used to calculate the value of by the following equation: (6) (7) (8) According to Wang et al. [29], the value of is set to 1/3. 2.3 Estimation of dominant orientation To achieve rotation invariance, one or more dominant orientation(s) are estimated for every feature. The orientations are estimated by considering a local region which is resampled to a circular area of radius . The local region is selected from the AAG filtered image. Then, PC model [16] is used to extract PC information from the selected circular area. As PC information is more robust against significant intensity variations, it is utilised in our method. The log Gabor filter responses [16] are analysed over different directions and scales to obtain the PC information [16]. The PC model [16] is expressed by (7) and (8). Here, () denotes the spatial coordinates of the point in an image and l is the scale of the filter. is the amplitude at the scale layer l and represents the phase. denotes the weighted mean phase, is the estimated noise level, and is a small constant to avoid division by zero. According to Kovesi [16], the values of and l are set to 0.01 and 4, respectively. is the sigmoidal weighting factor and represents an enclosed quantity whose value remains same when it produces a positive value, and in other cases its value is zero [16]. The noise is estimated as (9)where and denote the mean and variance, respectively, of the Rayleigh distribution which describe the noise response [16]. According to Kovesi [16], the value of is 2. The PC features extracted from the local region are used to construct a correlation surface similar to the LSS. Initially, the sum of square differences (SSDs) are calculated between the centre patch (p) and the surrounding patches of the local region containing PC information. Then, an exponential function uses the SSDs values to construct the correlation surface. The correlation surface is defined as (10)where denotes acceptable variations of intensity and presents the maximum variance of the difference of all patches with respect to the centre patch p. Dominant orientations of the features are estimated by using this correlation surface. In DOBSS [25] method also, a correlation surface is constructed according to standard LSS to determine the dominant orientations of the features. In the cited method, the correlation surface is divided into 36 angular bins to generate the orientation histogram. Then, the dominant orientations are identified from the orientation histogram according to the standard SIFT method. The peaks which belong to the 80% (according to standard SIFT [3]) of the highest peak of orientation histogram are considered as dominant orientations. However, this approach produces many peaks which are generated from the relatively lower correlation values. These peaks still belong to 80% of highest peak. These orientations adversely affect the matching performance. To solve this problem, the power law transformation [32] is used in our proposed method to modify the values of the correlation surface. The modified correlation values are used to generate the orientation histogram. The power law transformation is defined as (11)where denotes the input value and is the corresponding output value. and are the positive constants. Generally, the value of is considered as 1 [32]. Fig. 3 shows the effects of on for the different values of . From this figure, it can be observed that the lower input values are mapped to comparatively higher values when . For , the low input values are reduced more than the high input values. So, in the proposed method, condition is utilised to significantly reduce the lower correlation values. As result, the peaks generated by the lower correlation values do not belong to the 80% of the highest peak and these are not considered as dominant orientations. Fig. 3Open in figure viewerPowerPoint Plots of versus for the different values of The process of the dominant orientation estimation is illustrated in Fig. 4. Fig. 4a shows the filtered local area. The area covered by the red circle is used to estimate the dominant orientation. The PC features extracted from the local area are shown in Fig. 4b. Fig. 4c shows the correlation surface constructed by using the PC features. Fig. 4d shows the modified correlation surface obtained by using power law. The angular division of the modified correlation surface is shown in Fig. 4e. Finally, Fig. 4f shows the orientation histogram constructed by the modified correlation surface. Fig. 4Open in figure viewerPowerPoint Steps to determine the dominant orientation (a) AAG filtered local region, (b) PC features, (c) Correlation surface, (d) Modified correlation surface, (e) Angular division of the modified correlation surface, (f) Orientation histogram 2.4 Construction of structural descriptor A structural descriptor is constructed for every feature by considering an AAG filtered local circular region. The radius of the local region is determined by the M UR-SIFT algorithm. In standard LSS, the structural descriptors are constructed by considering a local circular region of radius 20 px [23]. Therefore, in our proposed method, the local region is resampled to a circular area of radius 20 px to obtain scale invariance. The PC information is obtained from the local circular area by using the PC model. In [10] also, PC is used to construct LSS-based descriptor for multi-modal image registration. However, the method is applicable for the globally corrected images which do not have orientation and scaling differences. On the other hand, our proposed method is developed to register the optical and SAR images having significant geometric differences. The PC features extracted from the local circular area are used to construct the correlation surface. In our proposed method, two correlation surfaces are constructed. Generally, the correlation surface of the LSS-based structural descriptor is constructed by calculating the SSDs values between the centre patch and the surround patches of the local circular region. In our proposed method, the first correlation surface is constructed by calculating the SSDs between the centred patch and the surround patches and the second one is constructed by calculating the SSDs between the transpose of the centre patch and the surround patches. This approach significantly improves the distinctiveness of the structural descriptor. The constructed correlation surfaces have been rotated in the direction of dominant orientation. According to Sedaghat and Ebadi [25], the distinctiveness of the LSS-based descriptor can be improved by dividing the sorted correlation values into different ordinal bins (). So, in our proposed method, the correlation values of each correlation surface are sorted into a non-descending order and these are equally divided into ordinal bins according to their orders. Finally, the ordinal bins are divided into log-polar structure with number of angular bins and number of radial bins. The proposed descriptor is constructed by concatenating the highest correlation value obtained from each of the log-polar divisions. Fig. 5 shows an illustration of the construction process of the proposed descriptor. The filtered local area used to construct the proposed descriptor is shown in Fig. 5a. Fig. 5b shows the PC features extracted from the local area. Fig. 5c shows the two correlation surfaces and constructed by the proposed approach. Although the two correlation surfaces are looking similar, still the correlation values are different for the two surfaces. The correlation surface is constructed by computing the SSD values between the centre patch and the surrounding patches, whereas is constructed by computing the SSD values between the centre patch and the surrounding patches. Therefore, the correlation values are different for the two surfaces. Fig. 5d shows the oriented correlation surfaces and toward the direction of dominant orientation. The ordinal divisions of the correlation values for each of the correlation surfaces are shown in Fig. 5e. Finally, Fig. 5f shows the log-polar divisions of the ordinal bins. Fig. 5Open in figure viewerPowerPoint Construction process of the proposed descriptor (a) Filtered local region, (b) Extracted PC features, (c) Computation of the two correlation surfaces, (d) Oriented correlation surfaces, (e) Ordinal division of the correlation surfaces, (f) Log-polar division of the ordinal bins Finally, feature matching is performed between the features of the optical and SAR images by using nearest neighbour distance ratio (NNDR) method [3]. According to NNDR method, if the distance ratio of the first nearest neighbour to the second nearest neighbour is <0.8, it is considered as matching pair. However, the matching pairs obtained by NNDR method contain many outliers. The fast sample consensus (FSC) [33] algorithm is used to identify the correct matching pairs and these are used to estimate the transformation between the images. The affine transformation model is used to estimate the transformation. 3 Results and discussion 3.1 Selected data sets Six pairs of optical and SAR images [34-36] are selected for the analysis of the experimental results. Fig. 6 shows the data sets. The images of these data sets have significant intensity as well as geometric differences such as rotation and scaling. Moreover, these are captured at different times. The information of the data sets is provided in Table 1. In Table 1, enhanced thematic mapper plus (ETM+), advanced land observing satellite (ALOS), fine beam dual (FBD), thematic mapper (TM), phased array-type L-band SAR (PALSAR), and UK-DMC: United Kingdom-disaster and monitoring constellation. Table 1. Information on the selected data sets Set Image Sensor Size Resolution, m Date Area 1 optical image ETM+ band 5 600 × 600 30 26 June 2000 Campbell river SAR image ALOS PALSAR 600 × 600 30 08 March 2011 2 optical image ETM+ band 3 580 × 580 30 28 May 2002 Panama canal, Panama SAR image TerraSAR-X 500 × 500 35 07 December 2013 3 optical image TM band 4 600 × 600 30 30 July 1988 Campbell river SAR image ALOS FBD 850 × 850 20 05 June 2010 4 optical image UK-DMC2 550 × 550 22 12May 2012 Kyushu, Japan SAR image ALOS FBD 600 × 600 20 06 July 2008 5 optical image ETM+ band 3 475 × 475 30 23 May 1999 Bhatsa Dam, Mumbai SAR image ALOS PALSAR 475 × 475 30 15 July 2007 6 optical image TM band 3 600 × 600 30 30 July 1988 Chesapeake Bay SAR image ALOS PALSAR 650 × 650 30 31 January 2011 Fig. 6Open in figure viewerPowerPoint Selected data sets (left: optical image, right: SAR image) (a) Data set 1, (b) Data set 2, (c) Data set 3, (d) Data set 4, (e) Data set 5, (f) Data set 6 3.2 Quantitative assessment The following parameters are used for the quantitative analysis of the proposed method: (i) NCM: The match obtained by the FSC algorithm are considered as the correct match. The NCM value should be as high as possible. (ii) Correct matching rate (CMR): It is computed as the ratio NCM to the total match which includes correct match as well as incorrect match. The number of match obtained by the NNDR method is the total match. The CMR value should be as high as possible. (iii) Root mean square error (RMSE): The RMSE value (in pixels) is computed by following the method presented in [6]. Initially, 30 uniformly distributed corresponding tie points are selected from the reference and sensed images by using environment for visualising images [37]. Here, and denote the tie points coordinates of the reference and sensed images, respectively. The tie points are carefully selected to reduce the residual as low as possible [38]. The transformation estimated by a registration algorithm is applied to the selected tie points of the sensed image, and then the RMSE value is calculated. RMSE is defined as (12)where ( [38]) is the number of corresponding tie points and is the transformed form of . The transformed form is obtained by applying the transformation estimated by a registration algorithm to . Lower RMSE value indicates better position accuracy in registration. (iv) MI: It is the measurement of the statistical similarity between two images. The MI value should be as high as possible. 3.3 Parameter setting The value of the parameter is decided through the experimental results analysis of the six selected image pairs. Fig. 7 shows the effect of on CMR values for these image pairs. From this figure, it can be observed that initially CMR values increase with the increase in the value of . The reason is that the comparatively lower correlation values are significantly reduced with the increase in the value of . As a result, the peaks generated by these lower correlation values do not belong to the 80% of the highest peak of orientation histogram and these are not considered as dominant orientation. However, after a certain range of the value of , the value of CMR again decreases. The reason is that many higher correlation values are converted to lower values for a very large value of . Considering these facts, the value of is set to 3. Fig. 7Open in figure viewerPowerPoint Effects of on CMR (%) for different image pairs The four parameters to be set for the proposed descriptor are , , , and . Experiments are performed on the selected six image pairs to decide the values of these parameters. The average performance of the proposed descriptor for the different combinations of these parameters is