Machine‐learning regression in evolutionary algorithms and image registration

IET Image ProcessingVolume 13, Issue 5 p. 843-849 Research ArticleFree Access Machine-learning regression in evolutionary algorithms and image registration Constantinos Spanakis, Corresponding Author Constantinos Spanakis kspan@ics.forth.gr School of Mineral Resources Engineering, Technical University of Crete, Chania, Greece Institute of Computer Science, Foundation for Research and Technology-Hellas, Heraklion, GreeceSearch for more papers by this authorEmmanouil Mathioudakis, Emmanouil Mathioudakis School of Mineral Resources Engineering, Technical University of Crete, Chania, GreeceSearch for more papers by this authorNikos Kampanis, Nikos Kampanis Institute of Applied and Computational Mathematics, Foundation for Research and Technology-Hellas, Heraklion, GreeceSearch for more papers by this authorManolis Tsiknakis, Manolis Tsiknakis orcid.org/0000-0001-8454-1450 Institute of Computer Science, Foundation for Research and Technology-Hellas, Heraklion, Greece Department of Informatics Engineering, Technological Education Institute of Crete, Heraklion, GreeceSearch for more papers by this authorKostas Marias, Kostas Marias orcid.org/0000-0003-3783-5223 Institute of Computer Science, Foundation for Research and Technology-Hellas, Heraklion, Greece Department of Informatics Engineering, Technological Education Institute of Crete, Heraklion, GreeceSearch for more papers by this author Constantinos Spanakis, Corresponding Author Constantinos Spanakis kspan@ics.forth.gr School of Mineral Resources Engineering, Technical University of Crete, Chania, Greece Institute of Computer Science, Foundation for Research and Technology-Hellas, Heraklion, GreeceSearch for more papers by this authorEmmanouil Mathioudakis, Emmanouil Mathioudakis School of Mineral Resources Engineering, Technical University of Crete, Chania, GreeceSearch for more papers by this authorNikos Kampanis, Nikos Kampanis Institute of Applied and Computational Mathematics, Foundation for Research and Technology-Hellas, Heraklion, GreeceSearch for more papers by this authorManolis Tsiknakis, Manolis Tsiknakis orcid.org/0000-0001-8454-1450 Institute of Computer Science, Foundation for Research and Technology-Hellas, Heraklion, Greece Department of Informatics Engineering, Technological Education Institute of Crete, Heraklion, GreeceSearch for more papers by this authorKostas Marias, Kostas Marias orcid.org/0000-0003-3783-5223 Institute of Computer Science, Foundation for Research and Technology-Hellas, Heraklion, Greece Department of Informatics Engineering, Technological Education Institute of Crete, Heraklion, GreeceSearch for more papers by this author First published: 21 March 2019 https://doi.org/10.1049/iet-ipr.2018.5389AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Evolutionary algorithms have been used recently as an alternative in image registration, especially in cases where the similarity function is non-convex with many local optima. However, their drawback is that they tend to be computationally expensive. Trying to avoid local minima can increase the computational cost. The purpose of authors’ research is to minimise the duration of the image registration process. This paper presents a method to minimise the computational cost by introducing a machine learning–based variant of Harmony Search. To this end, a series of machine-learning regression methods are tested in order to find the most appropriate that minimises the cost without degrading the quality of the results. The best regression method is then incorporated in the optimisation process and is compared with two well-known ITK image registration methods. The comparison of authors’ image registration method with ITK concerns both the quality of the results and the duration of the registration experiments. The comparison is done on a set of random image pairs of various sources (e.g. medical or satellite images), and the encouraging results strongly indicate that authors’ method can be used in a variety of image registration applications producing quality results in significantly less time. 1 Introduction Conventional mathematical methods of optimisation have a serious disadvantage when difficult optimisation problems need to be dealt with, especially when the objective function's nature is unknown. Since these methods need a good initial point that will lead to the global optimum, in such cases it is difficult, if not impossible, to find one. On the other hand, evolutionary algorithms ([1-3]) do not require to know the objective function's nature in order to optimise it. In addition, they do not require calculating the derivative of the functions that need optimisation, which can be helpful in cases where it is very difficult to calculate. Last but not least, they are usually based on fairly simple concepts such as natural phenomena or evolutionary concepts. This facilitates both their implementation as well as the development of novel algorithms and/or their hybridisation with other efficient algorithms. As a result, evolutionary algorithms can be generally adapted to any problem that can be modelled as a function. For these reasons, evolutionary algorithms like all meta-heuristic methods have long been used in solving problems that cannot be solved by conventional mathematical methods. The general pattern of evolutionary algorithms is this: starting from an initial random population of candidate solutions while new ones are created through recombination and mutation of previous ones. The latter are replaced by the new solutions and the procedure is repeated until a termination criterion is met. At the end, the best solution found so far will be considered as the optimum of the objective function. Even so, evolutionary algorithms have their own distinct disadvantages. One disadvantage common to all evolutionary algorithms is the repeated function evaluation for every new solution [4] that is generated. The total number of function evaluations needed to find a good approximation of the global optimum may be as small as a few hundreds and escalate to several thousands, which can be computationally expensive. Especially in the case, where even a single function evaluation can be computationally expensive, the computational cost skyrockets in the effort to avoid local optima, which is a problem that frequently occurs in intensity-based image registration methods. In order to solve the problem, many solutions have been proposed such as parallelism ([5-7]) and hybridisation ([8-10]) of evolutionary algorithms with other algorithms (heuristic or conventional). Another approach to minimise as much as possible the computational cost is to use approximation models ([11, 12]). By using models that create an approximation of the original function/problem which is simpler and faster to evaluate or easier to solve, we can reduce the cost of function evaluation and, in total, the computational cost of finding the global optimum. Here, we present a method of image registration using fitness approximation based on machine learning. Although, machine learning has been used widely in image registration process ([13-20]), to the best of our knowledge, no one has ever combined stochastic optimisation and machine-learning regression methods in intensity-based image registration methods. In the experiments, what is studied is its ability to register the images correctly, independently from the initial population and in relatively short time. More analytically, in the second section, we present our initial method for image registration without fitness approximation. Next, we present fitness approximation schemes using machine-learning regression techniques. In the fourth section, we introduce fitness approximation using machine learning in the evolutionary algorithm presented in the second section. Then, a series of experiments are presented along with the results before concluding with a discussion regarding fitness approximation. 2 Mutual information in image registration 2.1 Entropy Entropy [21] is a statistical measure of a random variable X, which describes its randomness and its mathematical formula is (1)where p(x) is the probability distribution of the random variable X. Apart from that, it shows how important the information we gain upon learning its value is. Imagine a grayscale image that has a unique colour (in our case the value of the pixel is our random variable X). Then, its entropy is minimum and learning its colour at a pixel is a trivial prediction, gaining no knowledge at all. On the other hand, learning a pixel value in a grayscale image that has many pixel values will certainly give us some information. In probability theory, we can define joint probability distribution of two or more random variables. In the same way, joint entropy of two or more random variable is defined: (2) 2.2 Mutual information Now that we have seen entropy, we can define mutual information [21]: (3)Its value shows how mutual the two random variables are, i.e. whether the knowledge of the first variable can give us information about the second variable and vice versa. In image registration, mutual information is used widely as a similarity measure ([22-24]) between two images. The idea is that when two images are aligned, then their mutual information is maximised, while when they are misaligned, their mutual information is decreased. Its ability to register two images with (probably) different modalities without the need to find specific features that are common in both images (something that can be very difficult especially when there are no discernible features) has spawned a series of successful registration methods. Its greatest disadvantage is that in order to calculate the joint probability distribution and the marginal probability distributions, we need to use a sizeable portion of each image, if not the whole of it, thus each evaluation of mutual information of images r and T(f) (the former is the reference or target image and the latter is the transformed/floating image) is computationally expensive. In order to reduce the computational cost in evolutionary computation, two traditional approaches have been proposed. One of them is Problem Approximation ([25-27]). The idea is to transform a complex problem into a simpler one, which may be easier to solve. The other one is fitness approximation ([11, 12], [25-28]). The idea behind this approach is to create an approximation of the original objective function using data samples of the form . Then, we create an approximation that is simpler to calculate, thus reducing the time needed to find the fitness of a candidate solution. There are many methods used for fitness approximation, such as polynomial models [27], neural networks [27], and support vector machines [29]. 2.3 Algorithm In the following Algorithm 1, we present the pseudo-code of our initial method which has been previously reported in [30]. This method is used here as a baseline. The parameters are the following ones: POPSIZE: The size of the population. N: Number of elements of the candidate solution. MAXITERS: The maximum number of iterations. : The minimum and the maximum value of parameter h, which is the possibility to choose a note from the harmonies of our memory. : The minimum and the maximum value of parameter par, which is the possibility to search for a note at the area of the best value using the update method of ALOPEX [31]. : The minimum and the maximum values of the notes, respectively. f: The function we seek to optimise. The algorithm starts by producing an initial random population (whose size is POPSIZE) of candidate solutions. Then, for MAXITERS iterations, a new candidate solution is produced. The value of the N elements of the new solution are calculated as follows: (4)where k and are randomly chosen candidate solutions of the population, is the new solution, is the current best solution and is the jth element of the solution. The first case () is applied with probability h, (5)where is the iteration and it is useful in exploiting the solutions of the population. Subsequently, the second case is applied only when the first case is applied and then again with probability par, (6)and its purpose is to exploit the area of the best candidate solution. The third and last case is applied with probability and its purpose is to explore the search space and avoid stumbling upon local optima. As we see in (5) and (6), the probabilities h and par are initially low so that more emphasis is given to exploration than in exploitation. As the method progresses, these parameters increase so that when the algorithm reaches the area of the global optimum, it will exploit it thoroughly in order to find it. Based on this method, the next section presents a number of possibilities to extend it by introducing an appropriate machine-learning approximation in order to minimise the execution time (Fig. 1). Fig. 1Open in figure viewerPowerPoint Algorithm 1 pseudo-code of original algorithm Machine-learning approximation experiments and results 3.1 Descriptions of the questions to address As we mention previously, evolutionary algorithms are able to overcome problems upon which conventional mathematical methods stumble. Their common disadvantage is the large number of fitness evaluation estimations that need to be done which makes them time-consuming. The idea is started using an approximation of the original function in order to partially replace the latter (i.e. from each 100 iterations of the algorithm in Algorithm 1 a percentage of them will use the original function in order to calculate the image similarity, while in the rest the approximation will be used). The questions we want to answer are the following ones: Which approximation method should we use for our problem? It must be of good efficiency regarding time and approximation. Can we reduce time without deteriorating the results? 3.2 Detailed description of the experiments and the results 3.2.1 Choice of machine-learning approximation method In our research context of multimodal image registration (i.e. aligning images derived from different modalities), we used mutual information as it is expressed by Mattes [32]. The rationale for using mutual information lies in its successful and wide use in image registration over the last two decades in a wide range of applications such as non-rigid in the biomedical domain [33] or optimising corner correspondences in computer vision [34]. There are other similarity measures that are also used in image registration such as cross-correlation coefficient [35] and normalised mutual information [36] on which our method can be applied as well. Our focus in this work, however, is on the optimisation and not on the objective function, i.e. the function we seek to optimise. In particular, the fact that mutual information computation takes a lot of time was the key factor that drove us towards the use of fitness approximation in evolutionary computation. The main factors taken into consideration as to which regression method should be used are the following ones: The unknown and peculiar form of image registration. For different pairs of images, mutual information has a different form and since there can be an infinite number of image pairs, there are infinite forms of mutual information graph. Its non-convexity and its non-linearity makes it hard to find a regression method that can approximate such functions. For a good fitness approximation, we need lots of data, which means that we need a method that is not only robust and accurate but also fast. A series of machine-learning regression methods from dlib [37], a C + + machine-learning library, were tested and compared with respect to mean absolute error as well as duration of training and evaluating. The methods used were the following ones: Kernel Recursive Least Squares algorithm (krls) [38] Kernel Ridge Regression (krr trainer) [39, 40] Multi-layer Perceptron (mlp) Radial Basis Function Network [41] (rbf_network_trainer) Epsilon-Insensitive Support Vector Regression with radial-basis kernel (svr trainer) [42] There is a plethora of additional approximation methods that could be tested such as multinomial regression [43], knn [44], random forest regression [45], as well as many variants using several kernels such as radial basis function and sigmoid kernel function [46]. In this work, we chose the abovementioned five methods from the widely used Dlib-ml library and the results were very satisfactory as it will be discussed in the next section. For this reason, there was no need for testing additional methods. In order to decide which approximation method we should use for our problem, we created a synthetic set of registration experiments. In order to conclude which machine-learning method is the most suitable for approximation, a randomly generated sample of 1500 image transformations was generated for each image pair The total number of pairs was 30 and were acquired from [47, 48, 49]. Then, each sample set was randomly partitioned 20 times in two subsamples. The first one had a thousand image transformations along with the corresponding mutual information and was used as the training sample, while the rest five hundred comprised the test sample. The training sample was then used for finding the approximation using each one of the aforementioned methods and the test sample for finding the mean absolute error of it for each partition i: (7)where f is the original function we want to approximate, N and are the size of the test set and the constructed approximation at the partition, and is an element of the test set. Having found the mean absolute error for each partition, we computed the mean absolute error (MEA) of each method as: (8)where is the MEA (7) of the partition. The indicative image pairs of Figs. 2 and 3 [47] and Figs. 4-6 [48] are used to illustrate our experiments. Fig. 7 presents the results of the approximation methods regarding the MEA. Fig. 2Open in figure viewerPowerPoint T2 transformation, (rotation, translation X, translation Y) = (-3.36E-01 radians, 5.34E + 00 pixels, 4.99E + 01 pixels) (a) T1 Source Image of Mouse Brain Slice 110, (b) T2 Target Image of Mouse Brain Slice 110 Fig. 3Open in figure viewerPowerPoint T2 transformation, (rotation, translation X, translation Y) = (-4.47E-01 radians, −2.15E + 01 pixels, −4.59E + 00 pixels) (a) T1 Source Image of Mouse Brain Slice 120, (b) T2 Source Image of Mouse Brain Slice 120 Fig. 4Open in figure viewerPowerPoint 78Greece_Fires transformation, (rotation, translation X, translation Y) = (-4.28E-01 radians, 3.83E + 01 pixels, 8.87E + 00 pixels) (a) 78 Greece Fires Source Image, (b) 78 Greece Fires Target Image Fig. 5Open in figure viewerPowerPoint 81 Oahe transformation, (rotation, translation X, translation Y) = (8.60E-03 radians, 1.54E + 01 pixels, 1.82E + 01 pixels) (a) 81 Oahe Reservoir Source Image, (b) 81 Oahe Reservoir Target Image Fig. 6Open in figure viewerPowerPoint 210_shrinking transformation, (rotation, translation X, translation Y) = (2.01E-01 radians, 4.59E + 01 pixels, −4.05E + 01 pixels) (a) Shrinking Lake Meredith Source Image, (b) Shrinking Lake Meredith Target Image Fig. 7Open in figure viewerPowerPoint Mean absolute error of machine learning function approximations As we can see in Fig. 7, radial basis networks and multi-layer perceptron had the worst MEA. Kernel recursive least squares and Kernel ridge regression produce much better results although, in the end, support vector regression (svr trainer), surpasses all other methods. In Fig. 8, we present the mean duration of the similarity approximation using -insensitive support vector regression (svr trainer). Considering its mean duration, which is almost close to zero, and its good approximation (compared to other presented methods of approximation), we chose -insensitive SVR as a surrogate function, i.e. a function that will partially substitute the computationally expensive mutual information, in order to reduce the duration of the image registration process. Algorithm 1 presents in detail our original algorithm, which will incorporate the machine-learning regression method. Fig. 8Open in figure viewerPowerPoint Mean duration of a single similarity estimation using -insensitive Support Vector Regression in all test images In the experiments regarding the original function combined with the SVR, we proceed according to Algorithm 1, albeit with a few modifications and adding another parameter si, i = 1,…,5, which is the size of the training sample which will be used for the construction of the approximation. In the beginning, we create a training sample equal to the initial population of random solutions and use the original mutual information in order to calculate the similarity of the images for the new candidate solutions, which are included in the training sample. The first training is applied when the training sample's size is equal to s1. Then, we start alternating between original function and approximation. We use the original function to create (POPSIZE-1) new solutions, which are then included in the training sample, and then (POPSIZE + 1), new solutions are evaluated using the approximation. These solutions are not included in the training sample. When the training sample's size becomes equal to s2, s3, s4, and s5, it creates new approximations, which can be more accurate due to the sample's larger size. The search stops when the maximum number of iterations is reached or the candidate solutions have reached a plateau where any more search is pointless. Next, we constructed a heterogeneous validation image pair dataset. From [47], we created 10 pairs of images using slices of registered T1 and T2 images from a mouse brain. The T1 images are retained as source images while the T2 images are randomly transformed and used as target images. For example, Mouse T2 110 image is initially co-registered with Mouse T1 110. Next, while the T1 image is left as it is, we transform Mouse T2 110 with (rotation, translation X, translation Y) equal to (-3.36E-01, 5.34E + 00, 4.99E + 01). In Table 1 (No 1–10), we show the transformations for each image. From [49], we obtained 10 images pairs, whose one image (which is randomly transformed (Table 1, No 11–20)) is a randomly transformed blurred version of the other. Last, from [48], we obtained 10 satellite aligned image pairs from 10 different areas, having for each one of them an image randomly transformed. In Table 1 (No 21–30), the random transformations used are presented. Table 1. Image transformation Transformation No Images Rotation Y translation X translation 1 Mouse T2 110 −3.36E-01 5.34E + 00 4.99E + 01 2 Mouse T2 120 −4.47E-01 −2.15E + 01 −4.59E + 00 3 Mouse T2 130 −4.45E-01 −1.53E + 01 −9.97E + 00 4 Mouse T2 140 −3.92E-02 −1.80E + 01 −4.69E + 00 5 Mouse T2 150 3.96E-01 9.22E + 00 3.71E + 01 6 Mouse T2 160 −3.97E-02 2.79E + 01 3.75E + 00 7 Mouse T2 170 4.52E-01 -3.11E + 01 3.85E + 01 8 Mouse T2 180 1.98E-01 4.89E + 01 −4.00E + 01 9 Mouse T2 190 −4.06E-01 −1.38E + 01 −3.09E + 01 10 Mouse T2 200 3.59E-01 4.92E + 00 −3.30E + 01 11 I01 4.39E-01 3.60E + 01 7.50E + 00 12 I02 1.77E-01 4.65E + 01 4.99E + 00 13 I03 −2.30E-02 2.71E + 01 −4.61E + 01 14 I04 1.12E-01 1.16E + 00 −2.81E + 01 15 I05 4.52E-01 1.20E + 01 −2.35E + 01 16 I06 2.91E-01 1.44E + 01 3.42E + 01 17 I07 1.75E-01 8.82E + 00 4.11E + 01 18 I08 −6.82E-02 4.99E + 01 −2.94E + 01 10 I09 −1.11E-01 4.49E + 00 −4.66E + 01 20 I10 −4.38E-02 −1.05E + 01 −4.56E + 01 21 78Greece_Fires −4.28E-01 3.83E + 01 8.87E + 00 22 81Oahe 8.60E-03 1.54E + 01 1.82E + 01 23 210_shrinking 2.01E-01 4.59E + 01 −4.05E + 01 24 Aguascalientes 3.78E-01 −4.38E + 01 6.90E + 00 25 BangongLake −8.41E-02 2.73E + 01 −3.58E + 01 26 Gibraltar 9.39E-02 −6.59E-01 −4.94E + 01 27 Greenland_Fires −3.81E-01 −1.20E + 01 3.99E + 01 28 LakeMead −4.26E-01 −1.83E + 01 −2.71E + 01 29 LakeShasta 3.61E-02 2.99E + 01 3.60E + 01 30 LakeUrmia −3.80E-01 2.30E + 01 1.09E + 01 This diversity in the image pairs used aims to ensure that the methods tested are invariant regarding specific applications (e.g. MRI or satellite image registration). 3.2.2 Introduction of SVR in the baseline method and its comparison with other image registration methods Having chosen the SVR as the best approximation method, the next step is to compare our proposed method with two well-known methods. The process of image registration is done using Mattes' mutual information from ITK [50] as a similarity metric. Our enhanced harmony search variant (EHAR) is compared with two image registration methods that are implemented in ITK (Insight Segmentation and Registration ToolKit) that use regular step gradient descent (RSGD) and one-plus-one evolutionary strategy ((1 + 1) ES) and the original harmony variant (HAR). We chose to compare HAR and EHAR with the aforementioned ITK methods, since ITK [50, 51] is one of the most widely known open-source image-processing toolkits that have been successfully used in many image processing applications (either as a dependency [52, 53, 54, 55] or as plugin [56]). The population size in HAR and EHAR is set to 40. A very small population size [57] can easily lead to premature convergence, due to lack of divergence, while a very large population can lead to slow convergence due to potentially too much divergence. The minimum and maximum values of parameters {h, p}, respectively, are set to{0,0} and {0.9, 0.4}. The initially low values of these parameters make the escape from local optima quite easier, and as we start to increase up to their respective maximum values, we shift from exploration to exploitation. Finally, the maximum number of iterations is set to 10,000. In the registration experiments, we used the same 30 image pairs as before (for assessing the most suitable machine learning method), but for each pair, only one transformation was evaluated which is reported in Table 1. Regarding the image pairs, each experiment for each pair was conducted 100 times for each optimisation method in order to test the reproducibility and efficiency of our method in speed and quality and compare it to that of the other methods. In Table 2, we present the number of the successful experiments for each method and pair. The larger the number, the more successful the method (given the image pair) is, with ‘0’ and ‘100’ rating the worst and the best performance, respectively. Table 2. Number of successful experiments No Experiment RSGD (1 + 1)ES HAR EHAR 1 Mouse T2 110 50 99 100 100 2 Mouse T2 120 46 60 100 100 3 Mouse T2 130 65 72 100 100 4 Mouse T2 140 72 100 100 100 5 Mouse T2 150 38 95 100 100 6 Mouse T2 160 69 100 100 100 7 Mouse T2 170 13 100 100 100 8 Mouse T2 180 16 100 100 100 9 Mouse T2 190 70 100 100 100 10 Mouse T2 200 89 100 100 100 11 I01 10 34 100 100 12 I02 5 100 100 100 13 I03 7 100 100 100 14 I04 55 100 100 100 15 I05 5 0 100 100 16 I06 3 49 100 100 17 I07 0 100 100 100 18 I08 0 94 100 100 19 I09 10 98 100 100 20 I10 0 100 100 100 21 78Greece_Fires 6 0 100 100 22 81Oahe 20 100 100 100 23 210shrinking 0 37 100 100 24 Aguascalientes 0 0 100 100 25 Bangong 64 84 100 100 26 Gibraltar 28 100 100 100 27 Greenland_Fires 0 0 100 100 28 Lake_Meadwater 31 75 100 100 29 Lake_Shasta 0 100 100 100 30 Lake_Urmia 45 100 100 100 As we see in Table 2, HAR and EHAR succeeded in all the experiments, with (1 + 1) ES being the third most successful and RSGD being the least successful one. The result indicates that our method is more robust in terms of accuracy and reproducibility due to the avoidance of local extrema, which is the main course of failure in such methods. To present our results regarding execution time, image pairs are grouped by image size. In Fig. 9, we see the mean execution time of the image registration for each optimisation method used with respect to the size (in pixels) of the image pairs used, while in Fig. 10, the mean duration of a single similarity evaluation in HAR and EHAR with respect to the size (in pixels) of the images used is presented (in the case of EHAR we want to see the impact of SVR in the mean duration of a single estimation, at the presence of the computational cost of approximation construction). There are some pairs of images of the same pixel size. Therefore, in Figs. 9 and 10, we calculate the mean time of image registration and single similarity evaluation for all these image pairs. This is the reason why there are fewer points (X-axis) in both figures. Fig. 9Open in figure viewerPowerPoint Mean execution time of the image registration process with respect to the size of the images in pixels Fig. 10Open in figure viewerPowerPoint Mean Duration of a single similarity estimation using -insensitive Support Vector Regression in all test images The duration in EHAR is reduced up to 44.44% compared to HAR and up to 40.64% compared to (1 + 1) ES, while