X‐Ray medical image super‐resolution via self‐organization neural networks and geometric directional gradient

IET Image ProcessingVolume 16, Issue 14 p. 3910-3928 ORIGINAL RESEARCHOpen Access X-Ray medical image super-resolution via self-organization neural networks and geometric directional gradient Khodabakhsh Ahmadian, Khodabakhsh Ahmadian orcid.org/0000-0003-3594-2762 Department of Electrical and Computer Engineering, Mahshahr Branch, Islamic Azad University, Mahshahr, IranSearch for more papers by this authorHamid-reza Reza-Alikhani, Corresponding Author Hamid-reza Reza-Alikhani Alikhani.hamid@aut.ac.ir Department of Electrical and Computer Engineering, Tafresh Branch, National University, Tafresh, Iran Correspondence Hamid-reza Reza-Alikhani, Department of Electrical and Computer Engineering, Tafresh Branch, National University, Tafresh 39518-79611, Iran. Email: Alikhani.hamid@aut.ac.irSearch for more papers by this author Khodabakhsh Ahmadian, Khodabakhsh Ahmadian orcid.org/0000-0003-3594-2762 Department of Electrical and Computer Engineering, Mahshahr Branch, Islamic Azad University, Mahshahr, IranSearch for more papers by this authorHamid-reza Reza-Alikhani, Corresponding Author Hamid-reza Reza-Alikhani Alikhani.hamid@aut.ac.ir Department of Electrical and Computer Engineering, Tafresh Branch, National University, Tafresh, Iran Correspondence Hamid-reza Reza-Alikhani, Department of Electrical and Computer Engineering, Tafresh Branch, National University, Tafresh 39518-79611, Iran. Email: Alikhani.hamid@aut.ac.irSearch for more papers by this author First published: 16 August 2022 https://doi.org/10.1049/ipr2.12603AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Nowadays, due to the rapid development of the Internet and communication networks, remote disease diagnosis by physicians utilizing medical images such as X-rays have increased. However, the transmitted images lose their true quality and resolutions due to several factors, including noise and image compression techniques used to conserve memory space. This paper presents a novel clinical X-ray image enhancement method based on self-organizing neural networks, the ability of first-order derivative operators to detect edges, and improved classification methods by enhancing image features and extending the nearest neighbor algorithms selection of the best similarity feature vector among all sample feature vectors. The proposed method employs first-order image derivation operators to extract edge details and features from images. The feature vectors were preprocessed and segmented into various classes using self-organizing neural networks. The best matching and most similar vectors among the feature vectors are retrieved via the nearest neighbor algorithm. In contrast to conventional and contemporary image super-resolution techniques, the proposed method does not rely on the cost function. It is independent of error backpropagation methods because the competitive nature of the neural network is used to update the weight vectors rather than the gradient descent method. In addition, the performance and accuracy of the proposed image super-resolution method were improved by increasing the number of feature components in the feature vectors through the use of the two-dimensional image gradient. Furthermore, numerous image quality measurement criteria, such as structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR), were used to compare the reconstructed images of the proposed algorithm with those of other image enhancement algorithms. The results show that the proposed image enhancement algorithms' high-resolution and high-quality reconstructed image significantly outperformed many other medical image enhancement algorithms in reconstructing the edges and boundaries of objects with a smooth slope, as deemed by expert personnel and compared with image quality measurement criteria. 1 INTRODUCTION X-rays are electromagnetic waves used to obtain images of internal organs, and the wavelength of these waves is extremely short. When this radiation enters the body, it is absorbed differently by various tissues based on their density. High-density bone absorbs significant radiation, whereas skin, fat, and muscle allows X-rays to pass unimpeded. The amount of radiation exposure necessary to obtain high-quality X-ray images of the body's internal organs must be increased, resulting in damage to the internal organs. Consequently, current X-ray imaging machines use low radiation doses to reduce the patients radiation exposure risk and rely on super-resolution imaging techniques to enhance image quality [1]. Medical images contain more intricate textures, details, and correlations than ground-truth images. In addition, the imaging equipment, the patients posture, involuntary movements, and the complexity of the patients body shape and flexibility can easily blur and alter medical images. Preparing high-quality, high-resolution medical images from patient bodies is one of modern medical sciences most significant challenges. X-ray imaging, on the other hand, is based on variations in the density and thickness of the tissues of the human body since the degree of absorption of each tissue layer varies when X-rays pass through the different tissue layers of the human body. As a result, the amount of X-rays reaching the screen or film of the imaging device varies. Thus, tissues with hardness structures absorb more X-rays and appear whiter, whereas softer tissues absorb fewer X-rays and appear darker. Physicians will correctly identify medical issues using X-ray images if they have access to a wealth of structural and specific information. The details of the edges and borders of the objects in reconstructed medical images are an essential structural element. For medical and clinical image enhancement, numerous super-resolution image methods have been proposed and are classified as learning-based, reconstruction-based, and interpolation-based methods based on their importance and effectiveness. Nearest neighbor, bilinear, and bicubic interpolation-based image super-resolution algorithms reconstruct high-resolution images from low-resolution images by predicting high-resolution pixels based on their neighbors in an oversampled grid. This method has low computational costs and high real-time performance. These simplistic methods tend to generate images with various effects near the edges and boundaries of the object, such as blurring and aliasing, as they are based on an oversimplified slow-variation image model [2]. Reconstruction-based image enhancement methods are employed to reconstruct high-resolution images. These techniques model a sequential system for image generation and frequently utilize such models to establish a linear system for high-resolution image reconstruction. This method restores images with excellent quality. Despite this, the computational cost is relatively high compared to its slow convergence time, its real-time performance is subpar, and as the image magnification increases, the image quality degrades. Moreover, the resulting image may be overly smoothed in many instances, with no discernible details at high frequencies [3]. By learning the statistical relationships between low- and high-resolution image patches across a large training dataset [4], learning-based methods can restore high-resolution images with sharp edges at a low computational cost. Image super-resolution techniques based on dictionary learning algorithms are referred to as shallow learning and deep learning [5]. Dong (2014) proposed a convolution neural network-based high-image super-resolution technique consisting of three layers of convolution neural networks for learning the nonlinear mapping relationship between low-resolution and high-resolution image patches. The study successfully reconstructed high-resolution images directly from low-resolution images Using the proposed method [6]. Image super-resolution based on convolution neural networks, on the other hand, depended on the context of small image regions and demonstrated a slow training time. As a result, the quality of the reconstructed image was inadequate, and details such as edges were obscured. To this end, they modified the previously proposed technique for super-resolution imaging by increasing the layer depth to eight layers, achieving greater speed and image quality than the three-layer technique. However, the network structure remained shallow, and image detail reconstruction was somewhat limited. Daoyu Lin et al. presented a rapid, accurate method for image enhancement using a recursive deep neural network that was both effective and efficient [7]. Compared to convolutional neural network-based image super-resolution methods that operate on the local receptive field, recursive neural networks were used to reduce the parameters of the neural network by increasing the size of the local receptive field to obtain more context information. However, the network layers were dense, and the training and reconstruction processes were time-consuming and complex [8]. These methods are ineffective for recreating high-quality X-ray images with larger dimensions than ground-truth images and require high-quality edge detail reconstruction [9]. Dean Huntrin et al. (2014) proposed a method for example-based medical image super-resolution and denoising that requires several time-consuming image preprocessing steps. These steps included capturing high-quality images of the internal organs, acquiring the necessary original low-resolution images of paired organs, and applying nonlinear programming techniques to generate sparse representations of the relationships between the high- and low-resolution images. Due to the backpropagation method used by their algorithm, the quality of the image reconstruction approach in medical image reconstruction was insufficient [10]. In another study, Yang Chen et al. (2014) proposed a super-resolution imaging method for improving medical images based on the K-SVD algorithm. The orientation and scale information of artifacts were used to train artifact atoms, and artifact and tissue feature atoms were combined to create three discriminative dictionaries. Based on these dictionaries, a discriminatory sparse rendering technique was used to eliminate streak artifacts. In order to minimize further noise and residual artifacts, a generic dictionary learning process was implemented to decrease additional noise and residual artifacts [11]. However, this method did not converge effectively and exhibited a high time complexity. Additionally, choosing an appropriate K value was challenging. Since the K-SVD algorithms output was not always globally optimal, the reconstruction quality of the method was insufficient. Chang Liu et al. have proposed using a random forest technique to identify the regression model with the lowest error rate for improving medical images. Training samples were clustered iteratively in this method, and then a series of standard regression models were developed. The clustered data were used to train a random forest model, which was then used to select the most effective regression model for image reconstruction. However, this method required training many regression models, which required significant storage space. In addition, for regression problems with high noise levels, random forest algorithms may not be able to reconstruct high-quality images from small image sets or feature vectors with a small number of features, and over-fitting may occur [12]. In the present study, we developed a new X-ray image enhancement method that eliminates most of the flaws and limitations of previous super-resolution techniques. In this method, we extracted image features based on the intensity of the image pixels and then reconstructed a single external dictionary. In order to eliminate random mathematical calculations in the programming structure of previous image super-resolution algorithms, we used a direct extraction of image features based on the intensity of the image pixel. We then reconstructed a single external dictionary to determine the intrinsic relationships between high- and low-resolution patches of images. Self-organizing neural networks are competitive neural networks that, unlike other networks, do not require the estimation of error functions or the backpropagation method to estimate the weights vectors of their neurons. In the proposed method, they were used to ignore the backpropagation error method when updating weight vectors in older image super-resolution methods. As a post-classifier, the k-nearest neighbor method was used to reduce the impact of image artifacts. Moreover, the gradient first-order image edge extraction filter was employed to improve the quality and sharpness of the image objects edges, which were specifications and requirements of medical images for physician diagnoses. In order to accelerate the proposed algorithms execution time, we employed the parallel programming method with hardware GPU. The proposed method reduced image noise and blur while improving image quality and resolution by enhancing the boundaries and edges of image objects in a smoothly sharpened image. 1.1 Motivations and contributions Currently, one of the most controversial images processing techniques is the single image super-resolution (SISR) method, which attempts to acquire high-resolution images from low-resolution images by estimating and approximating all missing high-frequency details, particularly in the case of clinical X-ray images, which require high-frequency information in addition to sharp edges with a smooth slope of the image objects in their structures. The vast majority of recently published work in image super-resolution has employed stochastic computation to determine the intrinsic relationships between high-resolution patches and their corresponding low-resolution patches in the dictionary. In addition, these methods cannot precisely estimate the cost functions quantity. Moreover, most neural network-based image enhancement techniques have employed the time-consuming backpropagation error method to compute and update the weight vectors of the networks neurons. Numerous prior methods for enhancing clinical X-ray images have incorporated tree search algorithms into their design. The tree search algorithms use statistical computation to identify the most suitable dictionary patches. Consequently, image super-resolution techniques utilizing tree search algorithms are neither precise nor robust. Using self-organizing neural maps [13] as a pre-classifier, nearest neighbor algorithm [14] as a post-classifier, geometric directional gradient edge detection operator as image edge extractor, and high-pass filter as a high-frequency image detail extractor, we propose a novel hybrid image super-resolution method to enhance the details of real ground-truth and clinical X-ray images. We trained, tested, and evaluated the proposed image super-resolution method and reconstructed the high-resolution image using a single external dictionary. Unlike other artificial neural networks that update their weight vectors via backpropagation, our proposed super-resolution method updates its weight vectors via competitive learning. Because the neural weight vectors are automatically adjusted and independent of manually adjusted error changes and fluctuations, it is more robust than most previous learning-based methods. In addition, there is no need to conduct statistical analysis or manually adjust the algorithms parameters. In addition, this method employs the nonparametric K-nearest neighbor algorithm to classify feature patch vectors based on their similarity. The high-frequency details of the image were enhanced by sharpening the edges and borders of objects in the high-resolution image with a gentle slope. Using CPU-GPU hardware technology and parallel programming, we improved the performance and execution speed of the proposed algorithm. We created the external dictionary, extracted the feature patch vectors, and reconstructed the high-resolution output image using CPU hardware and serial programming. During the reconstruction procedures, we also utilized GPU hardware and parallel programming for training the self-organizing neural network and the nearest neighbor algorithm and locating the high-resolution patch vector. The following are the main contributions of this paper: We used feature patch vectors and corresponding high-quality patches from low- and high-quality clinical X-ray images to populate these fields in an external dictionary with high- and low-quality feature patch vectors serving as dictionary field keys and high-quality patches serving as dictionary field values. After configuring the network parameters for the proposed neural network, the lattice weight vectors were set to values close to zero. After training the neural network with an external dictionary, the best matching neuron labels and the contents of each neuron based on vector titles were saved in a local table. We utilized the input low-resolution X-ray image in this study. In addition, the trained self-organizing network was used to reconstruct our high-resolution output image after preprocessing and processing. The images structure contained artifact effects, and its quality and resolution were inadequate. We utilized the nearest neighbor algorithm to mitigate the impact of image artifacts. This was achieved through the selection of the best-matching feature vectors. We evaluated our proposed super-resolution method to reconstruct high-quality images using clinical X-rays and ground-truth images. Several conventional and state-of-the-art super-resolution imaging algorithm magnification factors are compared to the outcomes. We incorporated GPU hardware and parallel programming into the proposed structure to increase the execution speed of the suggested algorithm for faster image reconstruction. The remainder of the paper is structured as follows: the relevant image enhancement procedures are described in Section 2. The proposed method for image enhancement is described in Section 3. Section 4 presents experimental results on multiple natural and X-ray image data sets to evaluate and assess the performance of the proposed method. The strengths and weaknesses of the proposed image super-resolution method are discussed in Section 5. The conclusion of the study is detailed in Section 6. 2 IMAGE ENHANCEMENT AND RELATED WORK 2.1 A typical medical imaging system model The forward acquisition model of the medical imaging equipment depicted in Figure 1 can be approximated by a discrete linear system similar to Equation (1). I k L R = D k P k W k I k H R + V k = H k ∗ I k H R + V k . \begin{align} I^{LR}_{k}= {D_{k}P_{k}W_{k}I^{HR}_{k}+V_{k}}={H_{k}*I^{HR}_{k}+V_{k}}. \end{align} (1)The low-resolution image I k L R $I^{LR}_{k}$ is a subsampled, blurred, and warped version of the high-quality image I k H R $I^{HR}_{k}$ . The P k $P_{k}$ -matrix, also known as a blur kernel or point spread function, is influenced by the parameters of the optical lens and imaging device. The W k $W_{k}$ and D k $D_{k}$ matrices represent warping and subsampling, respectively, of the kth HR image. In addition, parameter V k $V_{k}$ indicates the addition of Gaussian white noise. Therefore, combining the transformation matrices D k $D_{k}$ , W k $W_{k}$ , and P k $P_{k}$ into a single H k $H_{k}$ matrix is possible. The (*) symbol represents the convolution operation. The image super-resolution enhancement strategy may be ill-posed due to poorly trained, unfocused operators. Regularization is one of the techniques utilized to stabilize the inverse of an ill-posed problem. Using prior information, expressed as an unconstrained optimization problem, is the most effective way to solve the above ill-posed indeterminate system. Using maximum a posteriori estimation (MAP) methods, the ( I k H R $I^{HR}_{k}$ ) of the ( I k L R $I^{LR}_{k}$ ) recovery problem can be formulated by solving the optimization problem in Equation (2). In this equation, the σ parameter represents the white Gaussian variance. max exp − 1 2 σ 2 ∑ k = 1 K I k L R − H k I k H R 2 ] . \begin{align} \max \hspace{8.5359pt}\exp \left(-\frac{1}{2\sigma ^2}\sum ^{\rm K}_{\rm k=1}{\left(\left\Vert I^{LR}_{k}-H_{k}I^{HR}_{k}\right\Vert\right)^2}]\right). \end{align} (2)The regularization term is incorporated into the maximum estimate in several suggested regularization-based super-resolution methods. Equation (3) can be used to rewrite the problem of super-resolution optimization. In this equation, ( A I k H R ) $(AI^{HR}_k)$ is the regularization expression that specifies prior knowledge regarding the desired high-resolution image I k H R $I^{HR}_k$ . The regularization coefficient λ determines the regularization terms weight, while the parameter K relates to the number of images [15]. min 1 2 ∑ k = 1 K I k L R − H k I k H R 2 + λ A I k H R . \begin{align} \min \hspace{8.5359pt}\frac{1}{2}\sum ^{\rm K}_{\rm k=1}{\left(\left\Vert I^{LR}_{k}-H_{k}I^{HR}_{k}\right\Vert\right)^2} +\lambda {AI^{HR}_k}. \end{align} (3)We use Equation (4), the first component of the super-resolution optimization problem derived from Equation (3), to reconstruct the high-quality image with no regularization component and utilize the 2-norm to calculate the Euclidean distance between two vectors. min 1 2 ∑ k = 1 K I k L R − H k I k H R 2 . \begin{align} \min \hspace{8.5359pt}\frac{1}{2}\sum ^{\rm K}_{\rm k=1}{\left(\left\Vert I^{LR}_{k}-H_{k}I^{HR}_{k}\right\Vert\right)^2}. \end{align} (4) FIGURE 1Open in figure viewerPowerPoint Image degradation and reconstruction model 2.2 A geometrical directional gradient filter Through Equations (5) and (6), the spatially directional gradient derivative filter is a 2D gradient operator for estimating the first-order gradient in the geometric direction of the x and y coordinates. The parameter I in these equations represents the two-dimensional image. G x = ∂ I ( x , y ) ∂ x . \begin{align} G_{x}=\frac{\partial I(x,y)}{\partial x}. \end{align} (5)Likewise, we have G y = ∂ I ( x , y ) ∂ y . \begin{align} G_{y}=\frac{\partial I(x,y)}{\partial y}. \end{align} (6)As shown in Figure 2, the first-order derivative gradient operator can compute the derivative in the x and y directions. However, its computation is weaker at orthogonal vertices, So the Sobel operator has been used in this work to determine the Gradient Vector [ G x ${G}_x$ , G y ] T ${G}_y]{}^T$ , where G x ${G}_x$ and G y ${G}_y$ are the horizontal and vertical gradient components. Through Equations (7) and (8), this operator uses a 3x3 kernel to estimate the first-order gradient in 2D geometric space. This filter is the first spatially derivative filter that convolves with the input image to compute the images first-order gradient in the x- and y-axes. This kernels x- and y-direction edge widths are identical [16]. G x = + 1 0 − 1 + 2 0 − 2 + 1 0 − 1 ∗ I . \begin{align} G_{x}= \def\eqcellsep{&}\begin{bmatrix} +1&0&-1\\ +2&0&-2\\ +1&0&-1 \end{bmatrix}*I. \end{align} (7)Likewise, we have G y = + 1 2 + 1 0 0 0 − 1 − 2 − 1 ∗ I . \begin{align} G_{y}= \def\eqcellsep{&}\begin{bmatrix} +1&2&+1\\ 0&0&0\\ -1&-2&-1 \end{bmatrix}*I. \end{align} (8)In an image, the directional gradient operator highlights areas of intensity discontinuity, making it helpful in detecting small details and edges. The directional gradient operator is insensitive to noise effects, making it suitable for low-noise or smooth images. This first-order directional derivative operator is more expensive than the Laplace operator due to its two masks, additional edge direction information, and isotropic nature (equal weights in the x- and y- directions). FIGURE 2Open in figure viewerPowerPoint The images first-order derivative gradient in the x- and y- directions 2.3 Self-organizing maps organizational architecture approach In the 1980s, Teuvo Kohonen introduced Kohonen self-organizing maps, also known as Kohonen maps, as a novel artificial neural network. Unlike many other artificial neural networks, self-organized maps exhibit distinct structures and functions. Instead of backpropagation error techniques with stochastic gradient descent (SGD), the weight vectors of the networks neurons are adjusted using competitive learning and unsupervised methods. Self-organizing maps (SOM) [13] are a type of neural network consisting of a two-dimensional network in which the number of neurons varies based on the application of the neural network. A self-organizing neural networks lattice, also known as a feature map, has only two layers: one for input and one for output. Compared to the other types of neural networks, self-organizing maps lack an activation function in their structure. Therefore, the weight vectors are transmitted without intervention to the output layer. In self-organizing maps with the same dimension n as the input space, a weight vector has been assigned to each neuron of the lattice [17]. In self-organizing maps, neurons are connected by a connection specified by the neural networks topology, such as hexagonal network structures. These networks do not use backpropagation error techniques to update the weight vectors of self-organizing maps. As mentioned, this family of unsupervised artificial neural networks employs competitive learning algorithms instead. The three main characteristics of competitive learning are competition, cooperation, and adaptability. During the competition process, each neuron in the self-organizing maps is assigned a vector of initial weights with a value close to zero and the same dimension as the input space. Each neuron in the output layer has a vector of weights W=[ w 1 , w 2 , … , w n $w 1,w 2,\ldots,w n$ ] with dimension n in this study. At each iteration of the training step, self-organizing maps techniques calculate the distance between each neuron in the output layer and the input data sample P. As the competitions winner, the neuron with the shortest distance is selected. This winning neuron with the most similar vector components to the input vector P=[ p 1 , p 2 , … … … . . , p n $p 1,p 2,\ldots\ldots\ldots..,p n$ ] is referred to as the best matching unit (BMU). Typically, Euclidean metrics are utilized to calculate the distance between two feature vectors. When the BMU is successfully located during the cooperation step, the self-organizing maps algorithm updates the weights vector of the winning neuron and its neighbors during the final adaption stage. The Gaussian neighborhood function selects the winner neurons neighbors based on the self-organizing maps networks topology. Even for the winning neuron, the learning rate, which determines how much we wish to adjust our neural networks weights, will converge to zero after time t (positive infinity). Consequently, the weight vector of the winning neuron and the weight vectors of all its neighbors are modified, but not equally. Thus, the weight vectors of neurons located further away from the input vector are smaller. The algorithms terminating condition is a crucial processing parameter for neural network algorithms. If the stopping condition is closer to reality, the neural network will achieve its objectives more precisely. To this end, the stopping requirements for self-organized networks are that any variation in the weights vector must be zero, that is, the weights vector obtained in the preceding step must match the weights vector modified in the following step. This study's self-organizing neural networks training and testing processes are terminated using the same criteria. During the training phase of the self-organized maps algorithm, the pattern features vectors, such as pattern P feature vectors, are selected at random from the input patterns datasets. The self-organized maps algorithm for modeling tree search-based classification algorithms and classifying input feature vectors according to clustering methods is trained with all training feature vectors in two stages, coarse and fine. The distances between each feature vector and the networks weight vectors are then calculated using the Euclidean distance metric. The best matching unit is selected using the fewest possible functions. During the self-organized map test process, the weight vector of the winning neuron, which is very similar to the input feature vector, is used as the future representative of the classes. 2.4 Structure of the k-nearest neighbor algorithm In both classification and linear regression, the k-nearest neighbor algorithm is one of the most popular fundamental and essential algorithms for classification with the supervisor in machine learning. This algorithm is nonparametric, which means that, unlike Gaussian mixture models, it lacks fundamental knowledge about the input data's density and distribut