Medical image super‐resolution using correlation filter interleaved progressive convolution network (CFIPC)

Electronics LettersVolume 58, Issue 9 p. 360-362 LetterOpen Access Medical image super-resolution using correlation filter interleaved progressive convolution network (CFIPC) Ajay Sharma, Corresponding Author Ajay Sharma er.ajaysharma9303@gmail.com orcid.org/0000-0001-7951-9371 Electronics Department, Maulana Azad National Institute of Technology, Bhopal, India Email: er.ajaysharma9303@gmail.comSearch for more papers by this authorBhavana Prakash Shrivastava, Bhavana Prakash Shrivastava Electronics Department, Maulana Azad National Institute of Technology, Bhopal, IndiaSearch for more papers by this author Ajay Sharma, Corresponding Author Ajay Sharma er.ajaysharma9303@gmail.com orcid.org/0000-0001-7951-9371 Electronics Department, Maulana Azad National Institute of Technology, Bhopal, India Email: er.ajaysharma9303@gmail.comSearch for more papers by this authorBhavana Prakash Shrivastava, Bhavana Prakash Shrivastava Electronics Department, Maulana Azad National Institute of Technology, Bhopal, IndiaSearch for more papers by this author First published: 11 March 2022 https://doi.org/10.1049/ell2.12467AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract In medical image diagnosis, performance is affected because of degradation in image resolution, imaging equipment and imaging parameters. Currently, deep learning has gained much attention from researchers due to its capability to maintain perceptual quality after reconstruction. Therefore, this letter is motivated by the advantages of deep learning and proposes a novel model termed as the correlation filter interleaved progressive convolution network (CFIPC). In this letter, dilated convolution interleaved with a correlation filter expands the receptive field without any pixel information losses. The result analysis was performed on DRIVE, CHASEB1, MRI, ultrasound and histopathological dataset medical datasets at 2x/4x/8x/16x upscaling factors and achieved the highest PSNR/SSIM value of 50.54/0.9986 at a 2x upscaling factor on the histopathological dataset. Introduction Magnetic resonance imaging (MRI), computed tomography (CT), and ultrasound are examples of common medical imaging techniques used for expert diagnosis [1]. Due to hardware and existing imaging technology constraints, medical practitioners and researchers favour image super-resolution (SR) processing technology for medical diagnosis. Super-resolution is a technique for enhancing image features from a series of low-resolution (LR) images into a single high-resolution (HR) image. Medical imaging, data analysis and visualization, object identification, and other applications all benefit from super-resolution [2]. Super-resolution relies heavily on information gained from LR images for the reconstruction of SR images, and researchers [3] propose a Bayesian-based super-resolution approach for application in MR brain imaging. In [4], the authors introduce the random forest method, which does not need data training and converts LR patches to HR patches directly. Other techniques, such as the kernel estimate model [5] and the dictionary learning approach used in PET imaging [6], have proven accurate for single image super-resolution. But it requires complicated optimization to extract HR images and is time-consuming. In addition to that it requires manual tuning of parameters. In [7], the authors provided a remarkable study employing the Kalman filter for robust ultrasonic imaging that increased imaging performance. The use of filtering algorithms to obtain SR images necessitates many computational and storage load problems. Apart from that, the existence of blind spots in the SR images restricts the usage of such approaches. In [8], the authors developed enhanced Kalman filtering methods and a regression model to handle this problem. They employed an estimation theory-based framework with three kernel functions. Empirical findings imply that AISUKF with SK regression is better suited to the SR situation at hand, resulting in a denoised and detail-preserved output. Currently, because of the enormous capacity (payload) of the NN model and holistic learning, researchers are increasingly interested in using a neural network, a deep learning-based strategy, to solve the SR issue. These neural networks aid in acquiring previously applied functionality and the improvement of several deep learning techniques [9-11]. These upgraded deep neural network DNN techniques are cost-effective and provide appropriate quality while also cost-effective [12]. Coarse-to-fine SR CNN (CFSRCNN) was adopted in [13]. The proposed CFSRCNN uses a stack of multi-uses blocks to build a robust SR model. However, it has a poor convergence rate and a high training cost. Similarly, some researchers worked on residual networks for SR [13-17]. In [14], the authors presented the residual network based on the VDSR model. In this model, spatial location is extracted as an input feature. The model used the L1 loss function to train the model. The main drawback of this model was that it takes long training epochs to reach optimal results. In [17], the authors used the GAN model as the base model, as it is known that GAN results in more effective results, but it suffers from issues such as model collapse and training issues. However, training an SR model with such a large number of convolutions and parameters generally requires a high computational cost and greater memory storage. Similarly, in [13], the authors presented the dilated CNN architecture, but the activation function was ReLU, and the loss function was MSE. However, the occlusion problem was observed at a small scaling factor. This was removed by applying a progressive network with interleaved correlation filter [2] layers proposed in this letter. Therefore, this letter is motivated to develop super resolution for medical imaging systems. Methodology This letter is dedicated to designing an image super-resolution model for medical image diagnosis termed correlation filter (CF) interleaved progressive convolution network (CFIPC) for variable upscaling factors. The model architecture is presented in Figure 1. The model uses cascading recursive dilated convolution layers with a parametric-ReLU (pReLU) activation function and is termed dilated convolution PReLU network (DCPR). CFIPC used dilated convolution layers due to its capability to increase the receptive field in SR images. However, dilated convolution Convdimay cause some blind spots if used directly in SR applications [13]. Therefore, to resolve this issue, the letter designed the model, which slightly increases the kernel weight of the convolution layer by increasing the dilation factor. This leads to a linear increase in the receptive field without any loss in resolution. This is mathematically formulated as in Equation (1): C o n v d i = I ∗ d k t \begin{equation}Con{v_{di}} = I{*_d}{k_t}\end{equation} (1) Fig. 1Open in figure viewerPowerPoint Architecture of CFIPC where I is the discrete signal, kt is the kernel, * is the convolution, and d is the dilation factor. The recursive dilated convolution layers in CFIPC are formulated as Equation (2): k t ∗ x , α , k t ∗ 2 x , α , k t ∗ 3 x , α , k t ∗ 2 x , α , k t ∗ x \begin{equation}\left[ {{k_t}*x,\;\alpha ,{k_t}{*_2}x,\alpha ,{k_t}{*_3}x,\alpha ,{k_t}{*_2}x,\alpha ,\;{k_t}*x} \right]\end{equation} (2) where kt is the kernel weight, *n is the dilation factor, x is the receptive field, and αis the correlation filter factor. The output of each CFIPC layer is fed back to minimize losses that occurred while training. An ADAM optimizer performs the minimization of loss. The model is in the recursive stage until the minimum loss is not achieved. After reaching minimum loss, the output features are deconvoluted and passed on to the next recursive CNN layer. The minimized parameters are shared among recursive modules, and optimal parameters are passed from one layer to another. This increased the capacity of the super-resolution network and balanced the computational complexity. The loss function used here is mathematically represented as Equation (3): l o s s C F I P C = l o s s m s e + 1 − l o s s s s i m 2 \begin{equation}los{s_{CFIPC\;}} = los{s_{mse}}\; + \frac{{\left( {1 - los{s_{ssim}}} \right)}}{2}\end{equation} (3) l o s s m s e = ∑ i = 1 N ( x t − x r 2 N \begin{equation*}los{s_{mse}} = \frac{{\mathop \sum \nolimits_{i = 1}^N ({{\left( {{x_t} - {x_r}} \right)}^2}}}{N}\end{equation*} where xtis the target value, xris the reconstructed value, and N is the number of samples. l o s s S S I M = 1 2 1 − E o S S I M I L R , I ̂ H R \begin{equation*}los{s_{SSIM}} = \frac{1}{2}\;\left( {1 - {E_o}\left[ {SSIM\left( {{I_{LR}},{{\hat I}_{HR}}} \right)} \right]} \right)\end{equation*} where SSIM is the structural similarity, Eo[·] is the expectation operator over the spatial coordinates, and I L R , I ̂ H R ${I_{LR}},{\hat I_{HR}}$ is the LR image and reconstructed image, respectively. Algorithm 1. CFIPC Training Input: LRi = LR images; Training dataset, T d = { L R i n , H R i n } , where n ∈ s i z e ( T d ) ${T_d} = \{ {LR_i^n,HR_i^n} \}\;,\;{\rm{where}}\;n \in size( {{T_d}} )$ ;kt = kernel weight;*n = dilation factor;α= correlation filter factor Output: S R i n $SR_i^n$ , Super-resolution images. 1: For i = 1: epoch 2: Do 3: lossCFIPC 4: While lossCFIPC reaches convergence 5: end 6: H R i n = [ k t ∗ L R i , α , k t ∗ 2 L R i , α , k t ∗ 3 L R i , α , k t ∗ 2 L R i , α , k t ∗ L R i ] $HR_i^n = [ {{k_t}*L{R_i},\;\alpha ,{k_t}{*_2}L{R_i},\alpha ,{k_t}{*_3}L{R_i},\alpha ,{k_t}{*_2}L{R_i},\alpha ,\;{k_t}*L{R_i}} ]\;$ 7: if 8: lossCFIPC = min 9: Return H R i n $HR_i^n$ 10: end 11: end A deconvolution layer with a correlation filter (CF) is applied to optimize upscale the feature maps at the output of each CFIPC module. The training algorithm is represented in algorithm 1. Experimental results and discussion In this letter, we have conducted our experiments on medical image datasets, i.e., DRIVE [18], CHASEB1 [19], MRI [20], and ultrasound [21]. The proposed CFIPC SR network has been implemented on MATLAB 2020a with the MatConvNet package on Windows10. To train the CFIPC network, 150 training images are randomly taken from the DRIVE [18], CHASEB1 [19], MRI [20], and ultrasound [21] datasets. The training sample taken here is converted into image patches of size 48 × 48. The filter size was initially set to 128 and slightly decreased to 32. The CFIPC network is designed for image super-resolution with downsampling x/2, x/4, x/8, and x/16. The images are degraded using a bicubic down sampler by x/16 during training. The training is done for the 300 epochs. To test the model, DRIVE [18], CHASEB1 [19], MRI [20], ultrasound [21], and histopathological datasets [17] are used. The results are evaluated in terms of parameters such as the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), t-test observer, and naturalness image quality evaluator (NIQE). The high PSNR and SSIM show better reconstruction, whereas a smaller score of t-test and NIQE indicates better perceptual quality. Performance evaluation In this section, the letter presents the performance analysis of the CFIPC SR model. The model is tested on five datasets, as discussed above. In Table 1, we have presented the PSNR, SSIM, t-test, and NIQE evaluation on an input image for scale factors of 2x, 4x, 8x, and 16x, respectively. The HR image size for the DRIVE dataset is taken for testing as 1024×1024, as a reference with IGAN [16]. For the histopathological dataset, the input LR image size taken for testing is 96×96, with reference to SAI [17]. For other datasets, the input LR image was considered to be of size 128×128. The result shows that by introducing the dilated recursive model, pixel information [22] is preserved, which can be proven by achieving good performance evaluation values. Table 1. Parameter evaluation for the proposed CFIPC PSNR SSIM T Test NIQE s = 2 s = 4 s = 8 s = 2 s = 4 s = 8 s = 2 s = 4 s = 8 s = 2 s = 4 s = 8 DRIVE [18] 52.89 47.038 42.208 0.9952 0.984 0.9677 0.038 0.095 0.145 3.364 6.034 6.518 CHASEB1[19] 43.20 39.57 37.91 0.97 0.942 0.923 0.575 0.453 0.366 6.775 6.078 5.683 MRI[20] 32.605 29.491 29.007 0.9517 0.8804 0.8514 0.716 0.657 0.5590 2.999 6.55 5.787 Ultrasound[21] 41.782 40.396 40.30 0.9822 0.9635 0.966 0.6277 0.4179 0.2369 2.988 5.840 6.843 Histopathological dataset [17] 50.54 48.27 46.61 0.9986 0.9972 0.9972 0.0113 0.00005 0.0000021 18.80 8.41 7.419 Comparison with state-of-the-art This letter compares our model with different state-of-the-art methods for medical image super-resolution. We compare our model with other SR models, i.e., the multi-scale convolution neural network (MSCNN) [1], VDSR [14], SRGAN [15], improved generative adversarial networks (IGAN) [16], and SAI [17]. Tables 2–4 illustrate the comparative state-of-the-art. For state-of-the-art comparison, we have used two performance matrices such as PSNR and SSIM, as these matrices show the similarity structure between the ground truth and the reconstructed image. In Table 2, this letter compared the performance of VDSR, SRGAN, and IGAN on the DRIVE dataset. The result shows that IGAN has achieved a better PSNR/SSIM value, but with increased scaling factors, its perceptual quality was degraded. However, the proposed CFIPC shows improvement with increased scaling factors. In Table 3, the histopathological dataset was considered for evaluation. For this comparison, SAI [17] took the input LR image of size 96×96. Therefore, in this letter, we have also considered the LR image of size to be 96×96 and observed that CFIPC had shown better perceptual quality than VDSR and SAI. Similarly, in Table 4, this letter compared the result of ultrasound images with MSCNN [1] and observed better PSNR and SSIM. Figure 2 shows some reconstructed medical SR images of different images. The figure shows that CFIPC recovered more accurate results while preserving edges and textures. Therefore, it can be observed that the performance of CFIPC can achieve better perceptual quality for any medical image super-resolution. Table 2. Comparative state-of-the-art on the DRIVE dataset PSNR/SSIM Methods 4x 8x 16x Bicubic 46.41/0.9992 41.16/0.9975 37.11/0.9943 VDSR [14] 43.04/0.9991 37.92/0.9975 33.43/0.9940 SRGAN [15] 30.13/0.9741 26.43/0.9501 – IGAN[16] 47.78/0.9995 41.64/0.9982 38.00/0.998 OURS 47.038/0.984 42.20/0.9866 38.45/0.99 Table 3. Comparative state-of-the-art on the histopathological dataset PSNR/SSIM Methods 2x 4x VDSR [14] 30.288/0.9594 – SAI[17] 32.754/0.9509 30.314/0.8628 OURS 50.54/0.9986 48.27/0.9972 Table 4. Comparative state-of-the-art methods on the ultrasound dataset PSNR/SSIM Methods 2x 4x SRCNN[12] 40.4189/0.980 32.21/0.8858 MSCNN [1] 40.4213/0.982 32.53/0.90210 OURS 41.782/0.9822 40.396/0.9635 Fig. 2Open in figure viewerPowerPoint Examples of test results on different medical image datasets Conclusion In this letter, by embedding a correlation filter and recursive dilated convolution blocks in the CFIPC network and by using a new hybrid loss function, lossCFIPC, the effectiveness of the medical image SR model is presented. The best result on 2x upscaling, the DRIVE dataset, gives a PSNR value of 52.89, and at 4x and 8x, the histopathological dataset achieved 48.27 and 46.61, respectively. Similarly, the best SSIM value was observed on the histopathological dataset at 2x/4x/8x upscaling of 0.9986, 0.9972, and 0.9972, respectively. The state-of-the-art shows that CFIPC outperforms MSCNN, VDSR, SRGAN, IGAN, and SAI networks on parameters such as PSNR and SSIM. Moreover, this letter also evaluated other parameters, such as the t-test and NIQE, that show preservation of the perceptual quality of reconstructed SR images. CFIPC shows the significance of overcoming the smoothness problem and improving the results. Our model can be used for other images, such as SAR images, and for video super-resolution in future work. Conflict of interest The authors declare no conflict of interest. Funding information None Open Research Data availability statement None References 1Goyal, B., Dawa, Lepcha, C., Dogra, A., Wang, S.-H.: Correction to: A weighted least squares optimisation strategy for medical image super resolution via multiscale convolutional neural networks for healthcare applications. Complex Intell. Syst 7(6), 3325– 3325 (2021) CrossrefWeb of Science®Google Scholar 2Xia, R., Chen, Y., Ren, B.: Improved anti-occlusion object tracking algorithm using unscented Rauch-Tung-Striebel smoother and kernel correlation filter. J. King Saud Univ. - Comput. Inf. Sci. (2022). https://doi.org/10.1016/j.jksuci.2022.02.004Google Scholar 3Brown, K.G., Ghosh, D., Hoyt, K.: Deep learning of spatiotemporal filtering for fast super-resolution ultrasound imaging. IEEE Trans. Ultrason. Ferroelectr. Freq. Control, 67(9), 1820– 1829 (2020) CrossrefPubMedWeb of Science®Google Scholar 4Han, J., Wang, C.: Tsr-Tvd: Temporal super-resolution for time-varying data analysis and visualization. IEEE Trans. Vis. Comput. Graph., 26(1), 205– 215 (2020) PubMedWeb of Science®Google Scholar 5Chen, G., Wang, H., Chen, K., et al.: A survey of the four pillars for small object detection: Multiscale representation, contextual information, super-resolution, and region proposal. IEEE Trans. Syst. Man, Cybern. Syst., 52(2), 936– 953 (2020) CrossrefWeb of Science®Google Scholar 6Yamashita, K., Markov, K.: Medical image enhancement using super resolution methods. Lecture Notes on Computer Science (including Subser. Lecture Notes on Artificial Intelligence, Lecture Notes on Bioinformatics), 12141 LNCS, 496– 508 (2020) CrossrefGoogle Scholar 7Tang, S., Song, P., Trzasko, J.D., et al.: Kalman filter-based microbubble tracking for robust super-resolution ultrasound microvessel imaging. IEEE Trans. Ultrason. Ferroelectr. Freq. Control, 67(9), 1738– 1751 (2020) CrossrefPubMedWeb of Science®Google Scholar 8Kanakaraj, S., Nair, M.S., Kalady, S.: Adaptive importance sampling unscented Kalman filter with kernel regression for SAR image super-resolution. IEEE Geosci. Remote Sens. Lett., 19, 1– 5 (2022) CrossrefWeb of Science®Google Scholar 9Deeba, F., Kun, S., Ali Dharejo, F., Zhou, Y.: Wavelet-based enhanced medical image super resolution. IEEE Access, 8, 37035– 37044 (2020) CrossrefWeb of Science®Google Scholar 10Xue, S., Qiu, W., Liu, F., Jin, X.: Wavelet-based residual attention network for image super-resolution. Neurocomputing, 382, 116– 126 (2020) CrossrefWeb of Science®Google Scholar 11Xin, J., Li, J., Jiang, X., Wang, N., Huang, H., Gao, X.: Wavelet-based dual recursive network for image super-resolution. IEEE Trans. Neural Networks Learn. Syst., 33(2), 707– 720 (2020) CrossrefWeb of Science®Google Scholar 12Tian, C., Zhuge, R., Wu, Z., et al.: Lightweight image super-resolution with enhanced CNN. Knowledge-Based Syst., 205, 106235 (2020) CrossrefWeb of Science®Google Scholar 13Zhang, Z., Wang, X., Jung, C.: DCSR: Dilated convolutions for single image super-resolution. IEEE Trans. Image Process., 28(4), 1625– 1635 (2019) CrossrefPubMedWeb of Science®Google Scholar 14Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2016-December, 1646– 1654 (2015) Google Scholar 15Ledig, C., Theis, L., Huszár, F., et al.: Photo-realistic single image super-resolution using a generative adversarial network. Proc. - 30th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR 2017, 2017-January, 105– 114 (2016) Google Scholar 16Bing, X., Zhang, W., Zheng, L., Zhang, Y.: Medical image super resolution using improved generative adversarial networks. IEEE Access, 7, 145030– 145038 (2019) CrossrefWeb of Science®Google Scholar 17Chen, Z., Guo, X., Woo, P.Y.M., Yuan, Y.: Super-resolution enhanced medical image diagnosis with sample affinity interaction. IEEE Trans. Med. Imaging, 40(5), 1377– 1389 (2021) CrossrefPubMedWeb of Science®Google Scholar 18 DRIVE: Digital retinal images for vessel extraction. Available: http://www.isi.uu.nl/Research/Databases/DRIVE/. Accessed 17 Dec 2021 Google Scholar 19Fraz, M.M., Remagnino, P., Hoppe, A., et al.: An ensemble classification-based approach applied to retinal blood vessel segmentation. IEEE Trans. Biomed. Eng., 59(9), 2538– 2548 (2012) CrossrefPubMedWeb of Science®Google Scholar 20Rueda, A., Malpica, N., Romero, E.: Single-image super-resolution of brain MR images using overcomplete dictionaries. Med. Image Anal., 17(1), 113– 132 (2013) CrossrefPubMedWeb of Science®Google Scholar 21http://splab.cz/en/download/databaze/ultrasound. Accessed 17 Dec 2021. Google Scholar 22Chen, Y., Zhang, H., Liu, L., et al.: Research on image inpainting algorithm of improved total variation minimization method. J. Ambient Intell. Humaniz. Comput, 1– 10 (2021) PubMedWeb of Science®Google Scholar Volume58, Issue9April 2022Pages 360-362 FiguresReferencesRelatedInformation