# Multifocus Image Fusion in Q-Shift DTCWT Domain Using Various Fusion Rules

Authors: Yingzhong Tian; Jie Luo; Wenjun Zhang; Tinggang Jia; Aiguo Wang; Long Li
Journal: Mathematical Problems in Engineering
 (2016)

---

## Abstract

Multifocus image fusion is a process that integrates partially focused image sequence into a fused image which is focused everywhere, with multiple methods proposed in the past decades. The Dual Tree Complex Wavelet Transform (DTCWT) is one of the most precise ones eliminating two main defects caused by the Discrete Wavelet Transform (DWT). Q-shift DTCWT was proposed afterwards to simplify the construction of filters in DTCWT, producing better fusion effects. A different image fusion strategy based on Q-shift DTCWT is presented in this work. According to the strategy, firstly, each image is decomposed into low and high frequency coefficients, which are, respectively, fused by using different rules, and then various fusion rules are innovatively combined in Q-shift DTCWT, such as the Neighborhood Variant Maximum Selectivity (NVMS) and the Sum Modified Laplacian (SML). Finally, the fused coefficients could be well extracted from the source images and reconstructed to produce one fully focused image. This strategy is verified visually and quantitatively with several existing fusion methods based on a plenty of experiments and yields good results both on standard images and on microscopic images. Hence, we can draw the conclusion that the rule of NVMS is better than others after Q-shift DTCWT.


---

## Body

## 1. Introduction

Since the optical lens has limited depth of focus (DOF), it is difficult to obtain an image of an object with every part in the field focused. Due to the insensitivity of the human eyes to unsharpness or blurriness within the DOF of the lens, only the objects within the DOF appear to be in focus while objects out of the DOF are blurred [1]. The solution is to use multifocus image fusion to extract in-focus information from each partially focused image into one fully focused image, which is better for visual perception and further calculation. This technology has been extended to various fields, such as microscopic imaging, visual inspection, 3D shape recovery, and measurement [2]. Thus, it attracted the attention of lots of researchers, who presented a variety of fusion algorithms.Existing image fusion methods can be categorized into fusion methods on pixel, feature, and decision level [3]. In this paper, only pixel level methods are discussed due to their high accuracy and less information loss, which is better for microscopic imaging. Fusion methods on pixel level are subdivided into spatial domain and transform domain fusion algorithms [2, 4]. Transform domain algorithms, namely, the multiresolution algorithms, are more robust since the human visual system deals with information in a multiresolution way, which is in line with the processing principle of transform domain algorithms [5]. So, multiresolution algorithms can bring about higher precision, while spatial domain methods are commonly used in cases when rapidity is required. Transform domain fusion methods have two major research hotspots: pyramids algorithm [2, 6, 7] and DWT algorithm [2]. Presently, the DWT based fusion algorithm is a research hotspot with better fusion effect [5]. The DTCWT proposed by Selesnick et al. [8] is a relative improvement on DWT, with several important additional properties: approximate shift invariance, good directional selectivity, and limited data redundancy. In this Q-shift solution [8], the construction process of filters is largely simplified. Image fusion with guided filtering was presented by Li et al. [9]. Yin et al. presented a method of image fusion and superresolution using sparse representation [10]. Yang and Li reported an image fusion method on pixel level with simultaneous orthogonal matching pursuit [11].In this work, we have proposed an effective fusion rule for multifocus image fusion after using Q-shift DTCWT and performed this method over partially focused image sequences blurred by Gaussian operators and microscopic image sequences. The proposed method is compared with DWT and different pyramid algorithms. Some common objective metrics [12, 13] are used to analyze the performance of these algorithms and different fusion rules.The following renders the framework of this paper. Section2 deals with the preliminary details of Q-shift DTCWT. Section 3 describes the proposed approach of image fusion. Section 4 is about the fusion rules applied in the proposed approach. In Sections 5 and 6, the experimental results, performance comparisons, and time consumption are given. Finally, conclusions are drawn in Section 7.


## 2. Q-Shift DTCWT Related Theory

Aiming to solve the drawbacks caused by the DWT method, Kingsbury proposed Complex Wavelet Transform (CWT) method when conducting image fusion experiment. In the working process of CWT method, the plural form is needed, and perfect reconstruction filter banks should be built based on the plural form. It is relatively easy to construct such filter banks on the first decomposition layer, but it is much harder on more upper layers. In 1999, Kingsbury proposed a new form of DTCWT, which can both keep the advantages of CWT and achieve the function of perfect reconstruction [8]. DTCWT is a novel wavelet transform method using a pair of binary-structured trees to realize wavelet transforms on real and imaginary components of the signal in parallel. The principle of DTCWT decomposition process on a two-dimensional signal on level j is shown in Figure 1, while the reconstruction process is not presented as it is just an inverse transform of the decomposition one.Figure 1
Decomposition diagram of the Chinese character “wood” on levelj using Q-shift DTCWT.DTCWT algorithm gains a series of advantages due to its unique structure, which are approximate shift invariance, good directional selectivity, and limited data redundancy. Due to its shift invariance, images fused by DTCWT are smooth and continuous while images fused by DWT contain irregular edges. Another major superiority of DTCWT is good directional selectivity since DTCWT produces six subbands at each scale for both imaginary and real parts in (±15°, ±45°, ±75°), while DWT only presents limited directions in (0°, 45°, 90°), which improves the transform precision and keeps more detailed information. However, in DTCWT, the process of designing filters is a little complicated due to its requirement for satisfying simultaneously the biorthogonal conditions and the phase condition [8]. Q-shift DTCWT is a solution to improve this problem as its quarter shift filters produce complex wavelets that are exactly linear phase, which largely simplifies the filter construction process.The transform uses two parallel treelike filter series (Tree A and Tree B in Figure1, generating the real and imaginary parts, resp.) that realize complex wavelet filtering of the input signal. In order to achieve shift invariance, filters are designed to own different delays, ensuring that every filter can sample the values discarded by other filters because of downsampling. In this way, aliasing phenomenon is minimized. The trees are implemented on rows and columns of the signal, respectively, and generate two low frequency coefficients L(j+1,1) and L(j+1,2), containing rough information, and six high frequency coefficients H(j+1,m)(m=1,…,6), containing detailed information in different directions. In Figure 1, character “wood” is decomposed into eight subimages; two low frequency coefficients appear to be similar to the original one while the other six are the extracted detailed information in different directions. The two low frequency images are used as the input signal of the next level decomposition.


## 3. The Proposed Fusion Methods

The general image fusion scheme using DTCWT for both grayscale and color images is shown in Figure2. Since DTCWT can only be calculated on monochrome images, color images should be decomposed into subimages. As the commonly used RGB color model creates color distortion due to the high correlation among the R, G, and B components, we transform it into YIQ color model. Since Y (luminance) component contains much more information than I and Q (chrominance) components [14], we perform the DTCWT fusion only on Y components, while I and Q components are fused based on a mapping table corresponding to Y fusion. Finally, the images in YIQ color model are transformed in RGB color model for display. In case of dealing with grayscale images, the above steps are omitted and DTCWT fusion is performed directly on the source images.Figure 2
Schematic flowchart of the proposed fusion method.Detailed steps of DTCWT fusion process are as follows:(1)
Perform DTCWT decomposition process on the input images a to k, which are focused in different parts. Each image is decomposed into two low frequency coefficients (La1, La2) to (Lk1, Lk2) and some high frequency coefficients (Ha1, Ha2,…, Han) to (Hk1, Hk2,…, Hkn). The high frequency coefficients represent detailed information of the input image in different directions, and their number depends on the total decomposition level.(2)
Fusion rules are performed on low frequency coefficients and high frequency coefficients separately. More details are presented in the next section.(3)
The selected high frequency and low frequency coefficients are then adjusted with consistency check. In multifocus images, blurred and sharp areas are generally regionally connected. For example, if the surrounding pixels are in focus, then the single pixel inside must be in focus too. Consistency check [15] is aimed at picking and changing these single pixels to be consistent with surrounding ones.(4)
Finally, DTCWT reconstruction process is implemented on the final low frequency coefficients L1 and L2 as well as high frequency coefficients H1 to Hn to construct the fused image.


## 4. Fusion Rules

Fusion rule is the most important factor of the proposed fusion methods as it is the core of distinguishing focused region from defocused region. In the proposed method, low frequency coefficients and high frequency coefficients represent rough information and detailed information, respectively, requiring different fusion rules [16]. Several fusion rules are described in this section including pixel based fusion rules such as the Weighted Average (WA) method [17], the Synthesis image Module Value Maximum Selectivity (SMVMS), and regional based fusion rules like the Neighborhood Variant Maximum Selectivity (NVMS) [18], the Neighborhood Gradient Maximum Selectivity (NGMS) [19], and the SML [20]. Details of these fusion rules are discussed below.


### 4.1. The Weighted Average (WA) Method

WA is normally used in low frequency coefficients as differences in low frequency information are comparatively small. The final coefficientF(x,y) is defined as follows:(1)Fx,y=ω1p1x,y+ω2p2x,y+⋯+ωnpnx,y,where p1x,y to p2x,y are pixel values of source low frequency coefficients in the x,y location and ω1 to ωn are weights for p1x,y to p2x,y, generally ω1ton=1/n.


### 4.2. The Neighborhood Variant Maximum Selectivity (NVMS)

NVMS is defined as follows: calculate the standard varianceSTDnx,y of the pixel pnx,y in the nth picture in the x,y location: (2)STDnx,y=∑i=-N-1/2N-1/2∑i=-M-1/2M-1/2Pnx+i,y+i-Pn¯x,y2N×M,where N,M are the window sizes, generally N=M=3 or N=M=5. pn¯x,y is the average value which is given by(3)pn¯x,y=∑i=-N-1/2N-1/2∑j=-M-1/2M-1/2pnx+i,y+jM×N.The final coefficientFx,y is equal to the kth pixel pkx,y and k is defined by picking the biggest STDkx,y from all computed values. Hence,(4)STDkx,y=maxSTD1x,y,STD2x,y,…,STDnx,y.


### 4.3. The Neighborhood Gradient Maximum Selectivity (NGMS)

NGMS is defined byGnx,y which is formed as follows:(5)Gnx,y=∑i=-11∑j=-11Hi,jpnx+i,y+jH=-1-1-1-18-1-1-1-1.F x , y is equal to thekth pixel pkx,y and k is decided by the biggest Gkx,y. Hence,(6)Gkx,y=max⁡G1x,y,G2x,y,…,Gnx,y.


### 4.4. The Sum Modified Laplacian (SML) Method

The Sum Modified LaplacianSMLnx,y is given by(7)MLnx,y=2pnx,y-pnx-s,y-pnx+s,y+2pnx,y-pnx,y-s-pnx,y+sSMLnx,y=∑i=-N-1/2N-1/2∑j=-N-1/2N-1/2MLnx+i,y+j,where s denotes the step length and F(x,y) is thekth pixel pkx,y and k is defined by the biggest SMLnx,y. Hence,(8)SMLkx,y=max⁡SML1x,y,SML2x,y,…,SMLnx,y.


### 4.5. Synthesis Image Module Value Maximum Selectivity (SMVMS)

SMVMS is used in high frequency coefficients as high frequency information is generally bigger in focused area. In order to make the time consumption short, a synthesis imagepnix,y (i is the decomposition level) is formed by combination of coefficients in six directions, and the fused coefficient Fix,y is equal to the maximum pnix,y which is calculated as(9)pnix,y=∑i=16pni,jx,yFix,y=max⁡p1ix,y,p2ix,y,…,pnix,y.


## 4.1. The Weighted Average (WA) Method

WA is normally used in low frequency coefficients as differences in low frequency information are comparatively small. The final coefficientF(x,y) is defined as follows:(1)Fx,y=ω1p1x,y+ω2p2x,y+⋯+ωnpnx,y,where p1x,y to p2x,y are pixel values of source low frequency coefficients in the x,y location and ω1 to ωn are weights for p1x,y to p2x,y, generally ω1ton=1/n.


## 4.2. The Neighborhood Variant Maximum Selectivity (NVMS)

NVMS is defined as follows: calculate the standard varianceSTDnx,y of the pixel pnx,y in the nth picture in the x,y location: (2)STDnx,y=∑i=-N-1/2N-1/2∑i=-M-1/2M-1/2Pnx+i,y+i-Pn¯x,y2N×M,where N,M are the window sizes, generally N=M=3 or N=M=5. pn¯x,y is the average value which is given by(3)pn¯x,y=∑i=-N-1/2N-1/2∑j=-M-1/2M-1/2pnx+i,y+jM×N.The final coefficientFx,y is equal to the kth pixel pkx,y and k is defined by picking the biggest STDkx,y from all computed values. Hence,(4)STDkx,y=maxSTD1x,y,STD2x,y,…,STDnx,y.


## 4.3. The Neighborhood Gradient Maximum Selectivity (NGMS)

NGMS is defined byGnx,y which is formed as follows:(5)Gnx,y=∑i=-11∑j=-11Hi,jpnx+i,y+jH=-1-1-1-18-1-1-1-1.F x , y is equal to thekth pixel pkx,y and k is decided by the biggest Gkx,y. Hence,(6)Gkx,y=max⁡G1x,y,G2x,y,…,Gnx,y.


## 4.4. The Sum Modified Laplacian (SML) Method

The Sum Modified LaplacianSMLnx,y is given by(7)MLnx,y=2pnx,y-pnx-s,y-pnx+s,y+2pnx,y-pnx,y-s-pnx,y+sSMLnx,y=∑i=-N-1/2N-1/2∑j=-N-1/2N-1/2MLnx+i,y+j,where s denotes the step length and F(x,y) is thekth pixel pkx,y and k is defined by the biggest SMLnx,y. Hence,(8)SMLkx,y=max⁡SML1x,y,SML2x,y,…,SMLnx,y.


## 4.5. Synthesis Image Module Value Maximum Selectivity (SMVMS)

SMVMS is used in high frequency coefficients as high frequency information is generally bigger in focused area. In order to make the time consumption short, a synthesis imagepnix,y (i is the decomposition level) is formed by combination of coefficients in six directions, and the fused coefficient Fix,y is equal to the maximum pnix,y which is calculated as(9)pnix,y=∑i=16pni,jx,yFix,y=max⁡p1ix,y,p2ix,y,…,pnix,y.


## 5. Experimental Results and Analysis

The proposed algorithm has been operated on Matlab 2014a with an Intel i5 4590 processor with 4 GB RAM. The traditional fusion methods based on DWT [16] and pyramids algorithms such as Gradient Pyramid (GRAD), Morphology Pyramid (MOP), Ratio Pyramid (RAT), and Laplacian Pyramid (LAP) [2, 13] are presented as compared to the proposed one. The DWT method, pyramids methods, and the proposed Q-shift DTCWT methods are all decomposed on 5 levels. In the aspect of fusion rules, traditional methods all take the rule of low frequency WA and high frequency SMVMS. For the proposed one, five different fusion rules described in the last section are used, and different matches are shown in Table 1. DTCWT1 is for comparing the performances among Q-shift DTCWT and the traditional methods. The remaining four are improved methods with better performance. Low frequency information has high correlation with the adjacent ones, so it is natural to reach an inference that more regional based fusion rules could improve its performance.Table 1
Results of IQA indicators on fused images.
Matches Low frequency fusion rules High frequency fusion rules DTCWT1 WA SMVMS DTCWT2 NVMS SMVMS DTCWT3 NGMS SMVMS DTCWT4 SML SMVMS DTCWT5 NVMS NVMSIn DTCWT2 to DTCWT4, three regional based fusion rules are applied on low frequency coefficients to testify this inference and, in DTCWT5, the regional based NVMS is used both on low frequency and on high frequency coefficients for potential improvement.


### 5.1. Experiments on Standard Images

In the first experiment, the performance of the proposed fusion method is demonstrated by fusing 20 pairs of blurred images which are generated by filtering the source images shown in Figure3 with Gaussian filter. Due to space reasons, only six pairs are displayed but the others are similar to the presented ones. In each of these pairs, complementary regions of the source images are blurred. The source images are standard grayscale or color images and are taken as the ground truth images, serving as the templates for comparison.Figure 3
Standard grayscale and color images for testing. (a) Peppers. (b) Couple. (c) Airplane. (d) Balloon. (e) Lena. (f) Flowers.
(a) (b) (c) (d) (e) (f)We have found that it is difficult to evaluate the quality of fused images visually, especially when the differences are too small to be observed. Hence, objective methods are used for more scientific and accurate evaluation. The first objective evaluation method is pixel value subtraction. The difference image is obtained from subtracting ground truth image from the fused image. The subtraction results of “Peppers” are shown in Figure4 consisting of the ground truth image (Figure 4(a)), the partially blurred images (Figure 4(b) blurred in the middle Pepper and Figure 4(c) blurred in surroundings), subtraction results of the blurred images (Figures 4(d) and 4(e)), and subtraction result of Gradient Pyramid (Figure 4(f)), Morphology Pyramid (Figure 4(g)), Ratio Pyramid (Figure 4(h)), Laplacian Pyramid (Figure 4(i)), DWT (Figure 4(j)), and DTCWT1 to DTCWT5 (Figures 4(k)–4(o)).Figure 4
Subtraction results of “Peppers.” (a) The ground truth image. (b) Source image blurred in the middle Pepper. (c) Source image blurred in the surroundings. (d) Subtraction image of (b). (e) Subtraction image of (c). (f) Subtraction image with Gradient Pyramid. (g) Subtraction image with Morphology Pyramid. (h) Subtraction image with Ratio Pyramid. (i) Subtraction image with Laplacian Pyramid. (j) Subtraction image with DWT. (k) Subtraction image with DTCWT1. (l) Subtraction image with DTCWT2. (m) Subtraction image with DTCWT3. (n) Subtraction image with DTCWT4. (o) Subtraction image with DTCWT5.From analyzing the color of these subtraction images according to the color bar, we can judge the deviation between each fused image and the ground truth image. Obviously, subtraction results of GRAD and MOD bear relatively poor effect and those of RAT and LAP are better but perform extremely poor in edges. DWT performs better than the pyramids methods, and, by comparing DWT and DTCWT1, it can be concluded that DTCWT is better than DWT under the same fusion rules. But, with regional based fusion rules in low frequency coefficients, as in DTCWT2 to DTCWT4, the fused results are much better. Finally, in DTCWT5, where both low frequency and high frequency use regional based NVMS fusion rule, the deviation mainly focuses on the edge of the middle Pepper.Another objective evaluation method is called the quantitative metrics [21]. Quantitative metrics are the objective evaluation indicators which can overcome the influence of human’s inaccurate vision judgment and make the indicators mathematically evaluate the effectiveness of image fusion. Three quantitative metrics are applied here, including mutual information (MI), peak signal-to-noise ratio (PSNR), and root mean square error (RMSE), which are described as follows.


#### 5.1.1. Mutual Information (MI)

LethA,hB, and hF denote the normalized histogram of the source image A, source image B, and fused image, respectively. hAF and hBF denote the joint histogram between the fused and source images. P and Q denote the row size and column size of the image. Mutual information between source images A, B and resulting fused images is as follows:(10)MIA=∑i=1P∑j=1QhAFijlog2⁡hAFijhAijhFijMIB=∑i=1P∑j=1QhBFijlog2⁡hBFijhBijhFij.The total mutual information is defined as(11)MI=MIA+MIB.Larger values imply better image quality.


#### 5.1.2. Peak Signal-to-Noise Ratio (PSNR)

(12) P S N R = 10 log 10 ⁡ L 2 1 / P Q ∑ i = 1 P ∑ j = 1 Q R i j - F i j 2 ,where Rij, Fij denote, respectively, the (i,j)th pixel value of the reference image (ground truth image) and fused image and L is the count of gray levels in the image. PSNR value is higher if the ideal and fused images are more approximate. Higher value indicates better fusion.


#### 5.1.3. Root Mean Square Error (RMSE)

RMSE between the reference image and the fused image can be calculated as(13)RMSE=1PQ∑i=1P∑j=1QRij-Fij2.RMSE approaches zero when the reference and fused images are very approximate and it will increase when the similarity decreases.The values of MI, PSNR, and RMSE of the fused images with various fusion methods are shown in Tables2–4 and the best values are marked in bold. It can be seen that DTCWT2, DTCWT4, and DTCWT5 show better performance while being analyzed with MI. The PSNR values of LAP, DTCWT1, DTCWT2, and DTCWT3 methods are superior to the others. When calculated with RMSE, the best values belong to RAT, DTCWT3, and DTCWT4. Since DTCWT based fusion methods bear the best evaluation values in most situations, it can be concluded that the proposed fusion methods are superior to the existing pyramids and DWT methods while the fusion rules could be chosen in accordance with specific conditions, which will be studied in the future work.Table 2
Performance comparison of the standard images with MI.
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 40.16 5.10 35.68 6.41 39.72 6.66 MOP 39.71 4.29 35.36 5.74 40.36 6.03 RAT 46.73 6.24 40.35 7.78 46.41 7.49 LAP 47.12 6.02 42.52 7.65 46.46 7.33 DWT 47.32 6.44 42.36 7.80 46.30 6.69 DTCWT1 49.66 6.98 42.95 8.27 47.46 7.32 DTCWT2 51.09 7.06 42.46 8.42 47.63 7.31 DTCWT3 50.12 7.04 42.06 8.39 47.63 7.68 DTCWT4 51.07 6.98 41.84 8.48 47.86 8.01 DTCWT5 52.11 7.02 44.12 8.50 48.30 7.66 Pep.: Peppers; Cou.: Couple; Air.: Airplane; Bal.: Balloon; Flo.: Flowers. The abbreviations here also apply to Tables3, 4, and 9.Table 3
Performance comparison of the standard images with PSNR.
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 27.77 30.60 28.63 29.15 29.27 27.13 MOP 29.07 30.55 28.16 29.86 30.85 28.22 RAT 28.14 28.80 26.69 30.35 28.82 27.20 LAP 41.78 46.34 37.21 44.03 39.62 38.74 DWT 41.09 45.28 34.87 42.14 38.94 34.24 DTCWT1 45.62 51.64 37.29 45.42 40.65 35.49 DTCWT2 46.41 52.13 35.96 45.16 40.32 35.03 DTCWT3 46.45 52.07 36.04 45.19 40.32 34.90 DTCWT4 46.36 51.75 35.83 45.17 40.05 34.79 DTCWT5 45.02 47.93 34.94 43.54 38.86 34.14Table 4
Performance comparison of the standard images with RMSE.
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 6.15 4.75 5.27 5.75 5.33 6.11 MOP 5.42 3.27 4.95 4.96 4.59 5.55 RAT 0.20 1.09 1.22 1.39 1.24 2.23 LAP 1.67 0.89 2.16 1.07 1.34 2.05 DWT 1.61 0.89 2.46 1.32 1.70 2.75 DTCWT1 1.12 0.49 2.06 0.81 1.38 2.05 DTCWT2 0.91 0.44 2.11 0.71 1.40 2.05 DTCWT3 0.83 0.42 1.82 0.63 1.13 1.30 DTCWT4 0.93 0.48 2.13 0.64 1.59 1.16 DTCWT5 1.04 0.68 2.20 0.86 1.59 2.09


### 5.2. Experiments on Microscopic Images

In the second experiment, the fusion process is performed on two groups of images captured by a metallographic microscope. As described in Section1, only the objects within the DOF of the objective lens of the microscope appear to be in focus. In reality, observed specimens within the field sometimes have rough surfaces which could not be completely captured in the DOF in one shot. A method is proposed to solve the problem in which several pictures partially focused are captured by adjusting the height of the lens and hence different parts are within the DOF. Then, fusion methods presented above are applied on the image sequence to obtain an entirely clear image.Since metallographic microscopes are widely used in machinery, electronics, chip, chemical, precision instrument, and other industries for observing and analyzing surface quality of opaque matters, two examples, namely, the detection of Printed Circuit Board (PCB) and worn external turning tools, are presented in this paper with a metallographic microscope at 5x magnification and a CCD camera connected to a computer as our experimental devices. Fusion results are analyzed both visually and objectively.In the first PCB experiment, since the components and their joints are uneven, it is hard to capture a totally clear image in one shot. But, with the methods proposed in this article, completely focused images can be acquired. Three image sequences of PCB are shown in Figures5, 6, and 7, but, due to space reasons, only sequence 1 is displayed in detail. The PCB we used is shown in Figure 5(a) and the inspected area is marked in red blocks. In Figure 5, we can see three source images focused on board (Figure 5(b)), the solder paste (Figure 5(c)), and the component pin (Figure 5(d)) and ten fused images (Figures 5(e)–5(n)) using various fusion methods including GRAD, MOP, RAT, LAP, DWT, and DTCWT1 to DTCWT5. For better observation, a small part as shown in the red spotted line block in Figure 5(b) is magnified into Figures 5(o)–5(y), in which Figure 5(o) is extracted from the source image as a ground truth image and Figures 5(p)–5(y) are extracted from fused images. It is obvious that fused images with pyramids methods (Figures 5(p)–5(s)) show severe distortion. DWT method is better but contains lots of vertical light beams which do not exist in DTCWT method. DTCWT1 to DTCWT4 contain some degree of light dispersion but DTCWT5 shows perfect performance.Figure 5
Source images and fused images of sequence 1 for PCB. (a) Tested PCB. (b) Source image 1. (c) Source image 2. (d) Source image 3. (e)–(n) Fused images with various fusion methods. (o) The magnified ground truth image. (p)–(y) The magnified fused images with various fusion methods.
(a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) (m) (n) (o) (p) (q) (r) (s) (t) (u)  (v) (w) (x) (y)Figure 6
Source image sequence 2 for PCB.
(a) (b)Figure 7
Source image sequence 3 for PCB.
(a) (b)In objective evaluation, we simply use PSNR and RMSE as our quantitative metrics since MI could not be used because each experiment has more than two source images. As it is impossible to obtain a completely clear image, we cut clear parts of source images as the ground truth image as shown in the red blocks in Figures5–7. The fused images are also cropped at the same position for objective evaluation.Values of PSNR and RMSE of these areas are shown in Tables5 and 6 and the best values are marked in bold. It can be seen that DTCWT4 and DTCWT5 own the best performance when evaluated with PSNR while RAT, DTCWT3, DTCWT4, and DTCWT5 are better in RMSE evaluation. Generally, the results appear to be a good match to that in subjective evaluation. We can see that the proposed methods perform better. Fused images of the proposed methods reserve more details from source images and thus are clearer than other methods with less distortion. Using the proposed DTCWT fusion methods, we can obtain totally clear images of PCB for components failure detection, solder paste detection, and other applications.Table 5
Performance comparison of the PCB images with PSNR.
Method Sequence 1 Sequence 2 Sequence 3 Part 1 Part 2 Part 3 Part 1 Part 2 Part 1 Part 2 GRAD 19.53 22.95 19.15 33.79 27.77 18.63 23.13 MOP 24.79 17.93 18.00 31.03 24.08 18.32 18.32 RAT 20.85 14.80 17.01 28.39 22.35 19.12 19.12 LAP 30.56 25.22 22.35 35.20 30.72 22.65 25.03 DWT 34.10 22.19 24.56 39.72 31.73 23.93 25.64 DTCWT1 37.33 24.17 31.67 45.66 36.68 24.68 28.90 DTCWT2 42.83 23.33 29.40 45.40 33.82 25.17 22.26 DTCWT3 33.34 20.53 26.03 43.66 33.86 21.92 23.18 DTCWT4 48.01 25.35 31.77 45.76 50.58 56.62 21.24 DTCWT5 42.22 25.74 29.16 48.58 33.58 59.39 30.98Table 6
Performance comparison of the PCB images with RMSE.
Method Sequence 1 Sequence 2 Sequence 3 Part 1 Part 2 Part 3 Part 1 Part 2 Part 1 Part 2 GRAD 6.20 7.17 6.84 5.20 3.34 6.87 6.15 MOP 4.96 10.5 9.13 7.71 3.73 8.93 10.42 RAT 4.65 3.32 4.35 2.66 0.24 3.69 2.72 LAP 4.10 5.89 5.64 3.29 1.91 4.79 4.78 DWT 3.16 6.06 5.66 3.52 1.74 4.84 5.66 DTCWT1 2.18 4.63 2.72 2.32 0.94 4.35 4.54 DTCWT2 0.80 3.95 2.91 1.96 0.84 2.96 3.87 DTCWT3 0.61 3.48 2.09 1.06 0.65 2.57 2.96 DTCWT4 0.60 3.57 2.82 0.69 0.79 0.24 7.21 DTCWT5 1.25 5.33 4.20 2.84 0.74 0.19 4.32For the second cutting tool experiment, image sequences for a worn external turning tool are shown in Figures8–10 with the turning tool shown in Figure 8(a) and the inspected area marked in red blocks. Due to its worn surface, the turning tool seems to be partially focused and partially defocused in one image. We analyze the fusion results both subjectively and objectively, where fused images are not presented due to space reasons. The clear parts from different source images are cut as the ground truth images, as shown in the red blocks in Figures 8–10.Figure 8
Source image sequence 1 for external turning tool.
(a) (b) (c) (d)Figure 9
Source image sequence 2 for external turning tool.
(a) (b)Figure 10
Source image sequence 3 for external turning tool.
(a) (b)PSNR and RMSE values of the fusion results are shown in Tables7 and 8. We can draw the conclusion that DTCWT based methods also have outstanding performances in fusion of turning tool images.Table 7
Performance comparison of the turning tool images with PSNR.
Method Sequence 1 Sequence 2 Sequence 3 Part 1 Part 2 Part 3 Part 1 Part 2 Part 1 Part 2 GRAD 21.23 16.78 24.91 20.93 30.23 27.70 20.84 MOP 21.05 15.93 20.54 18.32 30.09 28.49 18.71 RAT 17.29 13.37 17.16 16.73 24.33 23.91 16.79 LAP 27.22 18.93 23.26 19.81 35.64 34.55 21.73 DWT 28.89 19.31 23.73 22.66 36.78 39.02 22.56 DTCWT1 32.11 20.81 29.32 23.77 38.64 42.55 26.14 DTCWT2 34.90 20.49 24.22 20.85 39.32 44.31 23.21 DTCWT3 29.06 15.88 24.10 18.97 35.70 39.50 23.17 DTCWT4 39.91 48.72 25.34 41.54 63.63 53.18 24.39 DTCWT5 30.17 28.01 35.08 54.15 59.81 55.06 27.46Table 8
Performance comparison of the turning tool images with RMSE.
Method Sequence 1 Sequence 2 Sequence 3 Part 1 Part 2 Part 3 Part 1 Part 2 Part 1 Part 2 GRAD 7.31 7.21 5.26 4.12 4.32 6.48 4.62 MOP 9.15 6.65 8.28 6.34 3.34 9.10 4.55 RAT 5.15 4.08 2.33 1.27 2.42 1.90 2.47 LAP 5.97 5.69 3.42 2.54 1.87 5.59 2.48 DWT 6.77 5.48 5.06 3.16 2.22 6.19 1.96 DTCWT1 6.20 4.39 2.24 2.80 2.01 4.22 1.34 DTCWT2 4.09 1.87 2.84 2.28 0.51 4.09 0.55 DTCWT3 3.71 2.16 1.82 2.21 0.89 3.44 0.57 DTCWT4 0.70 1.77 2.60 1.65 0.11 4.99 0.29 DTCWT5 4.73 1.76 2.80 0.34 0.18 5.93 0.31Table 9
Calculating time of the standard images (unit: s).
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 0.617 0.251 0.663 0.266 0.539 0.246 MOP 2.240 1.116 2.567 0.774 2.586 0.790 RAT 0.449 0.252 0.355 0.211 0.354 0.220 LAP 0.601 0.249 0.440 0.239 0.437 0.228 DWT 6.335 1.746 5.902 1.640 5.833 1.628 DTCWT1 10.82 2.874 10.39 2.763 10.35 2.753 DTCWT2 10.81 2.798 10.33 2.758 10.33 2.759 DTCWT3 10.87 2.763 10.36 2.770 10.35 2.747 DTCWT4 10.90 2.803 10.40 2.761 10.29 2.764 DTCWT5 10.98 2.834 10.54 2.800 10.45 2.790


## 5.1. Experiments on Standard Images

In the first experiment, the performance of the proposed fusion method is demonstrated by fusing 20 pairs of blurred images which are generated by filtering the source images shown in Figure3 with Gaussian filter. Due to space reasons, only six pairs are displayed but the others are similar to the presented ones. In each of these pairs, complementary regions of the source images are blurred. The source images are standard grayscale or color images and are taken as the ground truth images, serving as the templates for comparison.Figure 3
Standard grayscale and color images for testing. (a) Peppers. (b) Couple. (c) Airplane. (d) Balloon. (e) Lena. (f) Flowers.
(a) (b) (c) (d) (e) (f)We have found that it is difficult to evaluate the quality of fused images visually, especially when the differences are too small to be observed. Hence, objective methods are used for more scientific and accurate evaluation. The first objective evaluation method is pixel value subtraction. The difference image is obtained from subtracting ground truth image from the fused image. The subtraction results of “Peppers” are shown in Figure4 consisting of the ground truth image (Figure 4(a)), the partially blurred images (Figure 4(b) blurred in the middle Pepper and Figure 4(c) blurred in surroundings), subtraction results of the blurred images (Figures 4(d) and 4(e)), and subtraction result of Gradient Pyramid (Figure 4(f)), Morphology Pyramid (Figure 4(g)), Ratio Pyramid (Figure 4(h)), Laplacian Pyramid (Figure 4(i)), DWT (Figure 4(j)), and DTCWT1 to DTCWT5 (Figures 4(k)–4(o)).Figure 4
Subtraction results of “Peppers.” (a) The ground truth image. (b) Source image blurred in the middle Pepper. (c) Source image blurred in the surroundings. (d) Subtraction image of (b). (e) Subtraction image of (c). (f) Subtraction image with Gradient Pyramid. (g) Subtraction image with Morphology Pyramid. (h) Subtraction image with Ratio Pyramid. (i) Subtraction image with Laplacian Pyramid. (j) Subtraction image with DWT. (k) Subtraction image with DTCWT1. (l) Subtraction image with DTCWT2. (m) Subtraction image with DTCWT3. (n) Subtraction image with DTCWT4. (o) Subtraction image with DTCWT5.From analyzing the color of these subtraction images according to the color bar, we can judge the deviation between each fused image and the ground truth image. Obviously, subtraction results of GRAD and MOD bear relatively poor effect and those of RAT and LAP are better but perform extremely poor in edges. DWT performs better than the pyramids methods, and, by comparing DWT and DTCWT1, it can be concluded that DTCWT is better than DWT under the same fusion rules. But, with regional based fusion rules in low frequency coefficients, as in DTCWT2 to DTCWT4, the fused results are much better. Finally, in DTCWT5, where both low frequency and high frequency use regional based NVMS fusion rule, the deviation mainly focuses on the edge of the middle Pepper.Another objective evaluation method is called the quantitative metrics [21]. Quantitative metrics are the objective evaluation indicators which can overcome the influence of human’s inaccurate vision judgment and make the indicators mathematically evaluate the effectiveness of image fusion. Three quantitative metrics are applied here, including mutual information (MI), peak signal-to-noise ratio (PSNR), and root mean square error (RMSE), which are described as follows.


### 5.1.1. Mutual Information (MI)

LethA,hB, and hF denote the normalized histogram of the source image A, source image B, and fused image, respectively. hAF and hBF denote the joint histogram between the fused and source images. P and Q denote the row size and column size of the image. Mutual information between source images A, B and resulting fused images is as follows:(10)MIA=∑i=1P∑j=1QhAFijlog2⁡hAFijhAijhFijMIB=∑i=1P∑j=1QhBFijlog2⁡hBFijhBijhFij.The total mutual information is defined as(11)MI=MIA+MIB.Larger values imply better image quality.


### 5.1.2. Peak Signal-to-Noise Ratio (PSNR)

(12) P S N R = 10 log 10 ⁡ L 2 1 / P Q ∑ i = 1 P ∑ j = 1 Q R i j - F i j 2 ,where Rij, Fij denote, respectively, the (i,j)th pixel value of the reference image (ground truth image) and fused image and L is the count of gray levels in the image. PSNR value is higher if the ideal and fused images are more approximate. Higher value indicates better fusion.


### 5.1.3. Root Mean Square Error (RMSE)

RMSE between the reference image and the fused image can be calculated as(13)RMSE=1PQ∑i=1P∑j=1QRij-Fij2.RMSE approaches zero when the reference and fused images are very approximate and it will increase when the similarity decreases.The values of MI, PSNR, and RMSE of the fused images with various fusion methods are shown in Tables2–4 and the best values are marked in bold. It can be seen that DTCWT2, DTCWT4, and DTCWT5 show better performance while being analyzed with MI. The PSNR values of LAP, DTCWT1, DTCWT2, and DTCWT3 methods are superior to the others. When calculated with RMSE, the best values belong to RAT, DTCWT3, and DTCWT4. Since DTCWT based fusion methods bear the best evaluation values in most situations, it can be concluded that the proposed fusion methods are superior to the existing pyramids and DWT methods while the fusion rules could be chosen in accordance with specific conditions, which will be studied in the future work.Table 2
Performance comparison of the standard images with MI.
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 40.16 5.10 35.68 6.41 39.72 6.66 MOP 39.71 4.29 35.36 5.74 40.36 6.03 RAT 46.73 6.24 40.35 7.78 46.41 7.49 LAP 47.12 6.02 42.52 7.65 46.46 7.33 DWT 47.32 6.44 42.36 7.80 46.30 6.69 DTCWT1 49.66 6.98 42.95 8.27 47.46 7.32 DTCWT2 51.09 7.06 42.46 8.42 47.63 7.31 DTCWT3 50.12 7.04 42.06 8.39 47.63 7.68 DTCWT4 51.07 6.98 41.84 8.48 47.86 8.01 DTCWT5 52.11 7.02 44.12 8.50 48.30 7.66 Pep.: Peppers; Cou.: Couple; Air.: Airplane; Bal.: Balloon; Flo.: Flowers. The abbreviations here also apply to Tables3, 4, and 9.Table 3
Performance comparison of the standard images with PSNR.
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 27.77 30.60 28.63 29.15 29.27 27.13 MOP 29.07 30.55 28.16 29.86 30.85 28.22 RAT 28.14 28.80 26.69 30.35 28.82 27.20 LAP 41.78 46.34 37.21 44.03 39.62 38.74 DWT 41.09 45.28 34.87 42.14 38.94 34.24 DTCWT1 45.62 51.64 37.29 45.42 40.65 35.49 DTCWT2 46.41 52.13 35.96 45.16 40.32 35.03 DTCWT3 46.45 52.07 36.04 45.19 40.32 34.90 DTCWT4 46.36 51.75 35.83 45.17 40.05 34.79 DTCWT5 45.02 47.93 34.94 43.54 38.86 34.14Table 4
Performance comparison of the standard images with RMSE.
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 6.15 4.75 5.27 5.75 5.33 6.11 MOP 5.42 3.27 4.95 4.96 4.59 5.55 RAT 0.20 1.09 1.22 1.39 1.24 2.23 LAP 1.67 0.89 2.16 1.07 1.34 2.05 DWT 1.61 0.89 2.46 1.32 1.70 2.75 DTCWT1 1.12 0.49 2.06 0.81 1.38 2.05 DTCWT2 0.91 0.44 2.11 0.71 1.40 2.05 DTCWT3 0.83 0.42 1.82 0.63 1.13 1.30 DTCWT4 0.93 0.48 2.13 0.64 1.59 1.16 DTCWT5 1.04 0.68 2.20 0.86 1.59 2.09


## 5.1.1. Mutual Information (MI)

LethA,hB, and hF denote the normalized histogram of the source image A, source image B, and fused image, respectively. hAF and hBF denote the joint histogram between the fused and source images. P and Q denote the row size and column size of the image. Mutual information between source images A, B and resulting fused images is as follows:(10)MIA=∑i=1P∑j=1QhAFijlog2⁡hAFijhAijhFijMIB=∑i=1P∑j=1QhBFijlog2⁡hBFijhBijhFij.The total mutual information is defined as(11)MI=MIA+MIB.Larger values imply better image quality.


## 5.1.2. Peak Signal-to-Noise Ratio (PSNR)

(12) P S N R = 10 log 10 ⁡ L 2 1 / P Q ∑ i = 1 P ∑ j = 1 Q R i j - F i j 2 ,where Rij, Fij denote, respectively, the (i,j)th pixel value of the reference image (ground truth image) and fused image and L is the count of gray levels in the image. PSNR value is higher if the ideal and fused images are more approximate. Higher value indicates better fusion.


## 5.1.3. Root Mean Square Error (RMSE)

RMSE between the reference image and the fused image can be calculated as(13)RMSE=1PQ∑i=1P∑j=1QRij-Fij2.RMSE approaches zero when the reference and fused images are very approximate and it will increase when the similarity decreases.The values of MI, PSNR, and RMSE of the fused images with various fusion methods are shown in Tables2–4 and the best values are marked in bold. It can be seen that DTCWT2, DTCWT4, and DTCWT5 show better performance while being analyzed with MI. The PSNR values of LAP, DTCWT1, DTCWT2, and DTCWT3 methods are superior to the others. When calculated with RMSE, the best values belong to RAT, DTCWT3, and DTCWT4. Since DTCWT based fusion methods bear the best evaluation values in most situations, it can be concluded that the proposed fusion methods are superior to the existing pyramids and DWT methods while the fusion rules could be chosen in accordance with specific conditions, which will be studied in the future work.Table 2
Performance comparison of the standard images with MI.
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 40.16 5.10 35.68 6.41 39.72 6.66 MOP 39.71 4.29 35.36 5.74 40.36 6.03 RAT 46.73 6.24 40.35 7.78 46.41 7.49 LAP 47.12 6.02 42.52 7.65 46.46 7.33 DWT 47.32 6.44 42.36 7.80 46.30 6.69 DTCWT1 49.66 6.98 42.95 8.27 47.46 7.32 DTCWT2 51.09 7.06 42.46 8.42 47.63 7.31 DTCWT3 50.12 7.04 42.06 8.39 47.63 7.68 DTCWT4 51.07 6.98 41.84 8.48 47.86 8.01 DTCWT5 52.11 7.02 44.12 8.50 48.30 7.66 Pep.: Peppers; Cou.: Couple; Air.: Airplane; Bal.: Balloon; Flo.: Flowers. The abbreviations here also apply to Tables3, 4, and 9.Table 3
Performance comparison of the standard images with PSNR.
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 27.77 30.60 28.63 29.15 29.27 27.13 MOP 29.07 30.55 28.16 29.86 30.85 28.22 RAT 28.14 28.80 26.69 30.35 28.82 27.20 LAP 41.78 46.34 37.21 44.03 39.62 38.74 DWT 41.09 45.28 34.87 42.14 38.94 34.24 DTCWT1 45.62 51.64 37.29 45.42 40.65 35.49 DTCWT2 46.41 52.13 35.96 45.16 40.32 35.03 DTCWT3 46.45 52.07 36.04 45.19 40.32 34.90 DTCWT4 46.36 51.75 35.83 45.17 40.05 34.79 DTCWT5 45.02 47.93 34.94 43.54 38.86 34.14Table 4
Performance comparison of the standard images with RMSE.
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 6.15 4.75 5.27 5.75 5.33 6.11 MOP 5.42 3.27 4.95 4.96 4.59 5.55 RAT 0.20 1.09 1.22 1.39 1.24 2.23 LAP 1.67 0.89 2.16 1.07 1.34 2.05 DWT 1.61 0.89 2.46 1.32 1.70 2.75 DTCWT1 1.12 0.49 2.06 0.81 1.38 2.05 DTCWT2 0.91 0.44 2.11 0.71 1.40 2.05 DTCWT3 0.83 0.42 1.82 0.63 1.13 1.30 DTCWT4 0.93 0.48 2.13 0.64 1.59 1.16 DTCWT5 1.04 0.68 2.20 0.86 1.59 2.09


## 5.2. Experiments on Microscopic Images

In the second experiment, the fusion process is performed on two groups of images captured by a metallographic microscope. As described in Section1, only the objects within the DOF of the objective lens of the microscope appear to be in focus. In reality, observed specimens within the field sometimes have rough surfaces which could not be completely captured in the DOF in one shot. A method is proposed to solve the problem in which several pictures partially focused are captured by adjusting the height of the lens and hence different parts are within the DOF. Then, fusion methods presented above are applied on the image sequence to obtain an entirely clear image.Since metallographic microscopes are widely used in machinery, electronics, chip, chemical, precision instrument, and other industries for observing and analyzing surface quality of opaque matters, two examples, namely, the detection of Printed Circuit Board (PCB) and worn external turning tools, are presented in this paper with a metallographic microscope at 5x magnification and a CCD camera connected to a computer as our experimental devices. Fusion results are analyzed both visually and objectively.In the first PCB experiment, since the components and their joints are uneven, it is hard to capture a totally clear image in one shot. But, with the methods proposed in this article, completely focused images can be acquired. Three image sequences of PCB are shown in Figures5, 6, and 7, but, due to space reasons, only sequence 1 is displayed in detail. The PCB we used is shown in Figure 5(a) and the inspected area is marked in red blocks. In Figure 5, we can see three source images focused on board (Figure 5(b)), the solder paste (Figure 5(c)), and the component pin (Figure 5(d)) and ten fused images (Figures 5(e)–5(n)) using various fusion methods including GRAD, MOP, RAT, LAP, DWT, and DTCWT1 to DTCWT5. For better observation, a small part as shown in the red spotted line block in Figure 5(b) is magnified into Figures 5(o)–5(y), in which Figure 5(o) is extracted from the source image as a ground truth image and Figures 5(p)–5(y) are extracted from fused images. It is obvious that fused images with pyramids methods (Figures 5(p)–5(s)) show severe distortion. DWT method is better but contains lots of vertical light beams which do not exist in DTCWT method. DTCWT1 to DTCWT4 contain some degree of light dispersion but DTCWT5 shows perfect performance.Figure 5
Source images and fused images of sequence 1 for PCB. (a) Tested PCB. (b) Source image 1. (c) Source image 2. (d) Source image 3. (e)–(n) Fused images with various fusion methods. (o) The magnified ground truth image. (p)–(y) The magnified fused images with various fusion methods.
(a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) (m) (n) (o) (p) (q) (r) (s) (t) (u)  (v) (w) (x) (y)Figure 6
Source image sequence 2 for PCB.
(a) (b)Figure 7
Source image sequence 3 for PCB.
(a) (b)In objective evaluation, we simply use PSNR and RMSE as our quantitative metrics since MI could not be used because each experiment has more than two source images. As it is impossible to obtain a completely clear image, we cut clear parts of source images as the ground truth image as shown in the red blocks in Figures5–7. The fused images are also cropped at the same position for objective evaluation.Values of PSNR and RMSE of these areas are shown in Tables5 and 6 and the best values are marked in bold. It can be seen that DTCWT4 and DTCWT5 own the best performance when evaluated with PSNR while RAT, DTCWT3, DTCWT4, and DTCWT5 are better in RMSE evaluation. Generally, the results appear to be a good match to that in subjective evaluation. We can see that the proposed methods perform better. Fused images of the proposed methods reserve more details from source images and thus are clearer than other methods with less distortion. Using the proposed DTCWT fusion methods, we can obtain totally clear images of PCB for components failure detection, solder paste detection, and other applications.Table 5
Performance comparison of the PCB images with PSNR.
Method Sequence 1 Sequence 2 Sequence 3 Part 1 Part 2 Part 3 Part 1 Part 2 Part 1 Part 2 GRAD 19.53 22.95 19.15 33.79 27.77 18.63 23.13 MOP 24.79 17.93 18.00 31.03 24.08 18.32 18.32 RAT 20.85 14.80 17.01 28.39 22.35 19.12 19.12 LAP 30.56 25.22 22.35 35.20 30.72 22.65 25.03 DWT 34.10 22.19 24.56 39.72 31.73 23.93 25.64 DTCWT1 37.33 24.17 31.67 45.66 36.68 24.68 28.90 DTCWT2 42.83 23.33 29.40 45.40 33.82 25.17 22.26 DTCWT3 33.34 20.53 26.03 43.66 33.86 21.92 23.18 DTCWT4 48.01 25.35 31.77 45.76 50.58 56.62 21.24 DTCWT5 42.22 25.74 29.16 48.58 33.58 59.39 30.98Table 6
Performance comparison of the PCB images with RMSE.
Method Sequence 1 Sequence 2 Sequence 3 Part 1 Part 2 Part 3 Part 1 Part 2 Part 1 Part 2 GRAD 6.20 7.17 6.84 5.20 3.34 6.87 6.15 MOP 4.96 10.5 9.13 7.71 3.73 8.93 10.42 RAT 4.65 3.32 4.35 2.66 0.24 3.69 2.72 LAP 4.10 5.89 5.64 3.29 1.91 4.79 4.78 DWT 3.16 6.06 5.66 3.52 1.74 4.84 5.66 DTCWT1 2.18 4.63 2.72 2.32 0.94 4.35 4.54 DTCWT2 0.80 3.95 2.91 1.96 0.84 2.96 3.87 DTCWT3 0.61 3.48 2.09 1.06 0.65 2.57 2.96 DTCWT4 0.60 3.57 2.82 0.69 0.79 0.24 7.21 DTCWT5 1.25 5.33 4.20 2.84 0.74 0.19 4.32For the second cutting tool experiment, image sequences for a worn external turning tool are shown in Figures8–10 with the turning tool shown in Figure 8(a) and the inspected area marked in red blocks. Due to its worn surface, the turning tool seems to be partially focused and partially defocused in one image. We analyze the fusion results both subjectively and objectively, where fused images are not presented due to space reasons. The clear parts from different source images are cut as the ground truth images, as shown in the red blocks in Figures 8–10.Figure 8
Source image sequence 1 for external turning tool.
(a) (b) (c) (d)Figure 9
Source image sequence 2 for external turning tool.
(a) (b)Figure 10
Source image sequence 3 for external turning tool.
(a) (b)PSNR and RMSE values of the fusion results are shown in Tables7 and 8. We can draw the conclusion that DTCWT based methods also have outstanding performances in fusion of turning tool images.Table 7
Performance comparison of the turning tool images with PSNR.
Method Sequence 1 Sequence 2 Sequence 3 Part 1 Part 2 Part 3 Part 1 Part 2 Part 1 Part 2 GRAD 21.23 16.78 24.91 20.93 30.23 27.70 20.84 MOP 21.05 15.93 20.54 18.32 30.09 28.49 18.71 RAT 17.29 13.37 17.16 16.73 24.33 23.91 16.79 LAP 27.22 18.93 23.26 19.81 35.64 34.55 21.73 DWT 28.89 19.31 23.73 22.66 36.78 39.02 22.56 DTCWT1 32.11 20.81 29.32 23.77 38.64 42.55 26.14 DTCWT2 34.90 20.49 24.22 20.85 39.32 44.31 23.21 DTCWT3 29.06 15.88 24.10 18.97 35.70 39.50 23.17 DTCWT4 39.91 48.72 25.34 41.54 63.63 53.18 24.39 DTCWT5 30.17 28.01 35.08 54.15 59.81 55.06 27.46Table 8
Performance comparison of the turning tool images with RMSE.
Method Sequence 1 Sequence 2 Sequence 3 Part 1 Part 2 Part 3 Part 1 Part 2 Part 1 Part 2 GRAD 7.31 7.21 5.26 4.12 4.32 6.48 4.62 MOP 9.15 6.65 8.28 6.34 3.34 9.10 4.55 RAT 5.15 4.08 2.33 1.27 2.42 1.90 2.47 LAP 5.97 5.69 3.42 2.54 1.87 5.59 2.48 DWT 6.77 5.48 5.06 3.16 2.22 6.19 1.96 DTCWT1 6.20 4.39 2.24 2.80 2.01 4.22 1.34 DTCWT2 4.09 1.87 2.84 2.28 0.51 4.09 0.55 DTCWT3 3.71 2.16 1.82 2.21 0.89 3.44 0.57 DTCWT4 0.70 1.77 2.60 1.65 0.11 4.99 0.29 DTCWT5 4.73 1.76 2.80 0.34 0.18 5.93 0.31Table 9
Calculating time of the standard images (unit: s).
Method Images Pep. Cou. Air. Bal. Lena Flo. GRAD 0.617 0.251 0.663 0.266 0.539 0.246 MOP 2.240 1.116 2.567 0.774 2.586 0.790 RAT 0.449 0.252 0.355 0.211 0.354 0.220 LAP 0.601 0.249 0.440 0.239 0.437 0.228 DWT 6.335 1.746 5.902 1.640 5.833 1.628 DTCWT1 10.82 2.874 10.39 2.763 10.35 2.753 DTCWT2 10.81 2.798 10.33 2.758 10.33 2.759 DTCWT3 10.87 2.763 10.36 2.770 10.35 2.747 DTCWT4 10.90 2.803 10.40 2.761 10.29 2.764 DTCWT5 10.98 2.834 10.54 2.800 10.45 2.790


## 6. Time Consumption

This part discusses the calculation time of the above experiments. From Tables9 and 10, it can be concluded that calculating processes using pyramids algorithm cost very little time while the DWT algorithm costs a little more time, and the proposed Q-shift DTCWT methods cost the longest time. It seems that different fusion rules utilized in this paper have little to do with the costing time.Table 10
Calculating time of PCB images and external turning tool (ETT.) images (unit: s).
Method Images PCB 1 PCB 2 PCB 3 ETT.1 ETT.2 ETT.3 GRAD 14.33 8.535 11.45 5.736 5.791 5.745 MOP 94.60 56.42 75.19 36.67 36.84 36.67 RAT 6.436 3.874 5.137 2.608 2.582 2.695 LAP 10.12 6.071 8.043 4.039 4.061 4.031 DWT 48.62 48.49 47.89 44.72 44.85 44.94 DTCWT1 91.14 87.00 87.51 81.84 81.73 82.06 DTCWT2 101.1 84.64 87.52 81.72 81.71 82.34 DTCWT3 90.47 85.16 87.76 81.75 82.14 82.03 DTCWT4 90.09 85.77 87.50 82.16 82.36 82.32 DTCWT5 92.96 90.37 89.44 83.36 83.27 83.77


## 7. Conclusion

This paper provides an effective method for fusing various images by using Q-shift DTCWT. Since DTCWT is approximately shift invariant and has six directions: (±15°, ±45°, ±75°), more than in DWT, it could preserve more detail information and edge information of source images, and the Q-shift resolution simplifies its filter construction process. Since fusion rule is another significant factor in image fusion, five different fusion rules are presented in this article and evaluated both visually and objectively. From the plenty of experimental data, we can draw the conclusion that Q-shift DTCWT methods are a little better than some other multiresolution fusion methods (using the same fusion rules), but, with low frequency regional based fusion rules or both low frequency and high frequency regional based fusion rules, the DTCWT methods have outstanding performances. The proposed methods are used on microscopic image fusion including PCB and worn external turning tool, and the results turn out to be consistent with the ideal experiments. The proposed methods were proven to cost more time than pyramids algorithm and the DWT algorithm, and it seems that using different fusion rules will not add the costing time. Future work will be done for founding better fusion rules and discussing whether different fusion rules are appropriate for different situations.



---

*Source: 5637306-2016-11-10.xml*