Underwater image enhancement via LBP‐based attention residual network

IET Image ProcessingVolume 16, Issue 1 p. 158-175 ORIGINAL RESEARCH PAPEROpen Access Underwater image enhancement via LBP-based attention residual network ZhiXiong Huang, ZhiXiong Huang orcid.org/0000-0002-8596-2203 School of Information and Electronic Engineering, Shandong Technology and Business University, Yantai, China Institute of Network Technology, ICT(YANTAI), Yantai, ChinaSearch for more papers by this authorJinjiang Li, Corresponding Author Jinjiang Li lijinjiang@gmail.com orcid.org/0000-0002-2080-8678 Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Yantai, China Correspondence Jinjiang LI, Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Yantai 264005, China. Email: lijinjiang@gmail.comSearch for more papers by this authorZhen Hua, Zhen Hua orcid.org/0000-0003-1638-2974 School of Information and Electronic Engineering, Shandong Technology and Business University, Yantai, China Institute of Network Technology, ICT(YANTAI), Yantai, ChinaSearch for more papers by this author ZhiXiong Huang, ZhiXiong Huang orcid.org/0000-0002-8596-2203 School of Information and Electronic Engineering, Shandong Technology and Business University, Yantai, China Institute of Network Technology, ICT(YANTAI), Yantai, ChinaSearch for more papers by this authorJinjiang Li, Corresponding Author Jinjiang Li lijinjiang@gmail.com orcid.org/0000-0002-2080-8678 Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Yantai, China Correspondence Jinjiang LI, Co-innovation Center of Shandong Colleges and Universities: Future Intelligent Computing, Yantai 264005, China. Email: lijinjiang@gmail.comSearch for more papers by this authorZhen Hua, Zhen Hua orcid.org/0000-0003-1638-2974 School of Information and Electronic Engineering, Shandong Technology and Business University, Yantai, China Institute of Network Technology, ICT(YANTAI), Yantai, ChinaSearch for more papers by this author First published: 21 September 2021 https://doi.org/10.1049/ipr2.12341AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Owing to the influence of light absorption and scattering in underwater environments, underwater images exhibit color deviation, low contrast and detail blur, and other degradations. This paper proposes an underwater image enhancement method combining a residual convolution network, local binary pattern (LBP), and self-attention mechanism. The LBP operator processes the input underwater images. The LBP feature images and underwater images thus obtained constitute the network input. The network consists of three modules: a color correction module to remove the color deviation in underwater images, detail repair module to restore the integrity of details, and an LBP auxiliary enhancement module for global enhancement of image details. The correction and repair modules generate the correct color image and detailed supplement images, respectively. The final-result image is obtained by superpositioning the two generated images. The experimental results confirm that our method can reproduce the bright colors and complete details of the visual effect, showing a significant improvement over other advanced methods in quantitative evaluation. 1 INTRODUCTION With the improvement of underwater equipment, humans are becoming increasingly adept at exploring marine resources and underwater cruises. The developments in autonomous underwater vehicle (AUV) [1] have added new application directions for machine vision [2]. The use of underwater robots eliminates the danger to underwater explorers and provides detailed ocean information using the underwater image sent back by the electronic detection equipment carried by the robot. However, underwater imaging often suffers from poor visibility and color deviation due to the propagation of underwater ambient light affected by the medium, absorption attenuation, and backscattering [3, 4]. As shown in Figure 1, the short-wavelength bluegreen light has less hue attenuation and can reach a greater depth than the long-wavelength redorange light. Therefore, most underwater images are dominated by blue and green colors [5, 6]. Absorption and scattering affect the quality of underwater images, leading to unsatisfactory underwater target detection and recognition. An underwater image enhancement method can enhance the contrast of the original image and reduce degradation caused by attenuation and scattering. Therefore, adding an underwater image enhancement preprocessing process before underwater target recognition and detection can help in the smooth development of underwater vision tasks. FIGURE 1Open in figure viewerPowerPoint Maximum depth of light with different wavelengths in water It is difficult to enhance underwater image. Because of the complexity of underwater imaging, there are serious color deviation and blur in underwater image. How to restore the color distortion underwater image to normal color is a complex problem. In addition, the underwater environment also contains a large number of sea creatures and scenery, and these important parts of the image contain a lot of detail information. Therefore, we hope to solve the problem of how to correct the impact of color deviation and solve the image blur problem without introducing artifacts and other color deviation, while protecting the details of the sea creatures and scenery itself as much as possible. This paper proposes an underwater image enhancement method using a residual convolution neural network combined with local binary pattern (LBP) and self-attention mechanism, the contributions of this paper are mainly reflected in the following four aspects: (1) We propose a multi-scale residual color correction network, which extracts features from images of different scales and corrects the color of underwater images through continuous residual blocks. (2) We propose a self-attention residual detail restore network, which generates a detailed restore matrix for the previous color correction image to enhance the network's attention to the defect part of the image. (3) We propose an LBP feature enhancement network to protect the damaged feature details due to the change of image scale. (4) In terms of visual quality and quantitative metrics, our method not only generates enhanced images that conform to human visual color, but also achieves outstanding scores in image quality evaluation standard. The rest of the paper is arranged as follows. Section 2 describes the related work in this field. In Section 3, we introduce the proposed underwater enhancement network, including the principle of self-attention mechanism and loss function. Section 4 introduces in detail the acquisition of the underwater image dataset, the comparison with the experimental results of four traditional methods and three deep learning methods, and the ablation study of related components. In Section 5, we draw conclusions and describe the future work. 2 RELATED WORK Underwater image enhancement can be roughly divided into traditional methods and deep learning methods. In this part, we explain the related work from these two aspects. 2.1 Traditional methods Li et al. [7] used blue-green channel dehazing and red channel correction to restore underwater images. Li et al. [8] performed underwater blurring and contrast enhancement based on the principle of minimum information loss and histogram prior distribution. Wang et al. [9] proposed an underwater image restoration method based on an adaptive attenuation curve prior, which uses the smoothness and attenuation of light to simulate the attenuation process of light in different underwater environments. Ancuti et al. [10] proposed a color channel compensation (3C) image enhancement method, which makes the opponent colors return to zero mean to restore the lost channel. Gao et al. [11] solve the problem of underwater image degradation based on the shape and function of the retina. Bai et al. [12] proposed an underwater image enhancement method based on global and local equalization of histogram and dual-image multi-scale fusion. Zhang et al. [13] proposed a method of color correction and bi-interval contrast enhancement. The color correction process uses a method based on sub-interval linear transformation. Liang et al. [14] use the attenuation map of each color channel to guide the color correction process of the underwater image, and then based on the multi-scale gradient domain detail enhancement method to compensate for the lost details of the image. These model-free methods have improved the visual quality of underwater images to a certain extent, but the results of enhancement do not completely remove the effect of color deviation, and will always introduce obvious other colors or be affected by artifacts. In the complex and changeable underwater environment, the ideal results cannot be produced as expected, which is precisely because they ignore the underwater imaging mechanism. Therefore, we have introduced an underwater image formation model while studying the deep learning method, so that the network can fully learn how to better restore the original color of things in the underwater environment. Inspired by the dark channel prior (UDCP) [15], Peng et al. [16] proposed a single image restoration method based on the generalized dark channel prior, which improved the image restoration method based on the discrete cosine transform. Peng et al. [17] used a method based on image blurriness and light absorption to estimate the depth of underwater scenes, and restored underwater images with the estimated depth and underwater imaging model. Akkaynak et al. [18] proposed a revised underwater imaging model, and they applied this revised model in [19] to reconstruct the color of underwater images. Zhou et al. [20] proposed a color-line model to deal with underwater scattering and absorption problems. Song et al. [21] established the first artificially annotated background light database, and proposed a background light estimation model and a transmission map estimation model to enhance underwater images. Berman et al. [22] extended the hazelines model to deal with wavelength dependent light attenuation and developed a new underwater dataset. These methods use physical models when restoring image colors, which has stronger robustness and generalization than model-free methods. However, due to the complexity of the parameters, most of these methods require a huge calculation process, and the effect on the specific underwater image is not good [23]. Our method takes into account the advantages of physical models and deep learning, making the process of underwater image enhancement faster and with good results. 2.2 Deep learning methods Deep learning has shown extraordinary results in computer vision [24-26], and related applications continue to appear in the field of underwater image enhancement. Hou et al. [27] proposed a joint transmission and image domains residual convolutional neural network for underwater image enhancement, and synthesized the paired dataset with the Jaffe-McGlamery underwater imaging model, and residual network obtained effectively removed the haze-effect. Under the influence of GAN, Li et al. first proposed WaterGAN [28]. In this method, the in-air color images and depth maps were used to generate paired realistic underwater images, which constitute the large training dataset for color correction of monocular underwater images. To overcome the limitations of WaterGAN in processing underwater images taken from unknown locations, Li et al. [29] proposed a weakly supervised adversarial model for underwater image color correction, which does not require paired underwater in-air image datasets. Yang et al. [30] proposed a conditional adversarial generation network with dual discriminators, which significantly improves the contrast of underwater images. Fu et al. [31] proposed to compensate the information with a global-local network and use compressed histogram equalization to further enhance the image quality. Li et al. [32] proposed an underwater synthesis algorithm based on underwater scene priors, which can generate underwater virtual images of different underwater types and degrees of degradation. In addition, Li et al. [33] provided a self-collected real paired underwater image dataset, and proposed a gated fusion network based on convolutional neural network. The network trained by this dataset showed good performance and generalization. Inspired by the physical model of underwater imaging, Li et al. [34] proposed an underwater image enhancement method guided by the percentage of the scene radiance reaching the camera. The method based on deep learning highlights the powerful color correction ability in underwater image enhancement, but this powerful correction ability may bring excessive enhanced results. When correcting the background color, the color of the thing itself is also destroyed. Compared with the traditional method, after the image features are processed by the deep network, the detailed++ information is not well enhanced, which makes the resulting results look unnatural. Therefore, in contrast to existing deep learning-based underwater image enhancement methods, our method has the following unique characteristics: (1) Multi-scale networks and continuous residual blocks are used to correct the color: we extract rich feature information on images of different scales, and alternately use convolutional layers and residual blocks with different numbers of convolution kernels, so as to improve the enhancement ability of the network for underwater images. Through continuous residual blocks, the content features and color information of images are organized and constructed. (2) Continuous residual block and self-attention mechanism are used to restore details: the network learns the residual information between the original image and the color-corrected image through the continuous residual block, so as to accurately generate more details from the region information near the defect point. We have also added a self-attention module to make the detailed image have more accurate environmental colors and richer and more detailed texture. (3) LBP features are used to enhance details: we also quoted LBP features to retain the detailed information of the image, which changes with the size of the image feature matrix in the network, and superimposes them in real time to protect the image details after the scale changes. 3 METHODOLOGY 3.1 Attention mechanism The attention mechanism originated from the visual attention system, which aims to simulate the performance of the primate visual system. Attention mechanisms have been greatly developed owing to the recent developments in deep learning. Bahdanau et al. [35] first applied the attention mechanism to natural language processing (NLP). Vaswani et al. [36] proposed a network framework that completely relies on the attention mechanism to deal with sequential problems. In addition to the NLP field, attention mechanism has also been widely used in computer vision [37-39].We try to apply the self-attention mechanism to underwater image enhancement task, and use the correlation between different image regions to enhance the network's ability to pay attention to internal features, so that the underwater image quality can be improved to achieve better results. As shown in Figure 2, we can see that the network with self-attention mechanism can calculate the global dependence between image regions. Therefore, it will pay more attention to the incompletely corrected regions and make the color correction of the image more thorough, which is very helpful for our underwater image enhancement task. FIGURE 2Open in figure viewerPowerPoint Self-attention effect comparison. (a) Raw underwater image, network result (b) without self-attention, (c) with self-attention We will introduce the self-attention module in detail (as shown in Figure 3), the input feature matrix x was input to 3 convolution layers with a kernel size of 1 × 1, the convolution layers output 3 multidimensional matrices f, g and h. f is used to query the influence of each point on the input pixel, g is the influence database of the query reference, and h is the matrix to retain the original channel information. Because module only focused on the degree of influence of the region on points, the channel numbers of f and g were compressed from 3 to 1 after linear transformation to reduce the computational complexity of the module, whereas h only experienced linear transformation and retained the original number of channels, and then performs the following matrix multiplication calculation [40]: s i , j = f ′ x i T · g ′ x j , (1)among them, f ′ ( x i ) T is the result of flattening the two-dimensional image information at position i of f into one-dimensional information and transposing it, and g ′ ( x j ) is the result of flattening the two-dimensional image information at position j of g into one-dimensional information, s i , j is equivalent to a dictionary that compares the position of i with position j. To calculate the weight of the influence of region j on the position of i, the reference dictionary uses the softmax operation as follows [41]: β i , j = exp s i , j ∑ 1 n exp s i , j (2)The β i , j obtained here represent the attention distribution matrix, which can give the degree of attention of the j th region to the i th position, and multiply it with h to obtain the self-attention feature matrix a t t [41]: a t t = a t t j ε R C , j = 1,2,…, N a t t j = ∑ i = 1 N β i , j h m i , h m j = W h x i , (3)where W h ε R C × N is the weight matrix obtained by network learning, and the attention of input features is redistributed at a t t . Next, to project the obtained a t t back to the input x, a t t is reshaped into that of x and then placed into the convolution layer with a convolution kernel of 1 × 1 for linear transformation to obtain a t t ′ . The final formula is as follows [38]: F f i n a l = γ · a t t ′ + x , (4)where γ is a learnable scalar initialized to 0. γ changes with an increase in the number of training epochs and gradually learns to assign more weight to non-neighboring regions to obtain a better attention characteristic matrix. Equation (4) finally superimposes the learned attention matrix back to the original image; we then feedback the detailed image to the residual network for the next synthesis process. 3.2 Local binary pattern LBP is a type of feature extractor that is simple in principle but very effective. The ability to obtain the global image features of the image by extracting its texture features makes it widely used in various computer vision tasks [42-44]. LBP has the characteristics of rotation and gray invariance. In addition, because LBP only compares the gray value of the center and region pixels, the regional gray value change caused by the illumination change has no noticeable effect on LBP. Therefore, LBP also has the characteristic of being insensitive to illumination changes. This means that the generated LBP image will not change due to the change of lighting conditions of the image area. In this paper, we use the illumination invariance of LBP and apply it to the underwater environment with uneven illumination, so that the neural network can obtain richer image feature input, and superimpose it in real time when the network gradually repairs the image color to improve the visual performance of enhanced images. From Figure 4, we can see that the LBP image of the original underwater image is basically the same as the LBP image of the enhanced image. We apply this invariance to the entire enhancement process of our network to make the underwater image enhancement task better. FIGURE 3Open in figure viewerPowerPoint Self-attention module structure. bs is the batch size; w and h are the width and height of the input image, respectively; ⊗ represents matrix multiplication; and ⊕ represents element summation FIGURE 4Open in figure viewerPowerPoint Comparison between the raw underwater image and result of our network enhancement and related LBP images. From top to bottom are the raw underwater images, LBP images of the raw underwater image, enhanced results, and LBP images of enhanced result The main concept behind LBP is comparing the gray value of a pixel with neighboring gray values. If the gray value of the neighboring pixel is greater than the gray value of the pixel, it is marked as 1; otherwise, it is marked as 0, and then an eight-bit binary number is obtained, which is usually converted to a decimal number to correspond to 256 kinds of LBP codes, and the obtained LBP codes are used to reflect the texture information of the area. In selecting the neighborhood, the early LBP adopted the surrounding 3 × 3 region, but this could not meet the image requirements of different sizes. Therefore, a circular LBP with a circular neighborhood was proposed to remove limitations in sample size and regions and is expressed as [45]: L B P P , R x , y = ∑ i = 0 P − 1 2 i S g i − g c , S x = 1, x ⩾ 0 0, x < 0 , (5)where L B P P , R ( x , y ) represents the LBP texture feature of the central pixel ( x , y ) , P represents the total number of sample pixels of the neighborhood, R is the radius of the neighborhood, g i is the gray value of the i th neighboring pixel, and g c is the gray value of the center pixel. After obtaining the corresponding LBP code according to each pixel position of the original image, the pixel values of the LBP feature map are composed of LBP codes. We put the LBP map into the network and superimpose it in real time when the image scale changes to protect the image details. 3.3 Network structure The images collected in an underwater environment suffer from color deviation and poor visibility. Therefore, the proposed network performs color correction and detail restoration on the original underwater image (as shown in Figure 5). For acquiring the image information more accurately, the network obtains more details of feature information on different scale images and alternates convolution layers and residual blocks with varying numbers of convolution kernels, thus improving the enhancement ability of the network for underwater images. The implementation specifics of our network is shown in Algorithm 1. Algorithm 1. Training of the network using SGD Algorithm 1 Algorithm of our network Input: Paired synthetic dataset: d s 1. Sample synthetic image I s and ground truth I t from d s 2. Generate LBP image: I L ( x , y ) = ∑ i = 0 P − 1 2 i S ( I i s − I c s ) 3. Generate correct color image: I c = f c ( I s ) When the size of Ic changes: I c = I c + f c ( I s ) 4. Generate detail supplement image: I d = f d ( I s , I c ) When the size of Ic changes: I d = I d + f d ( I s ) 5. Generate enhanced image: I e = I c + I d 6. Update network by minimizing the loss L f i n a l : L f i n a l = L m s e ( I e , I t ) + w × L 1 ( I e , I t ) In the color correction module, the input image first passes through a 64-kernel convolutional layer, the number of channels changes from 3 to 64, and then passes through two residual block regions. The residual block is the basic structure of our network. Figure 6 shows the internal composition of the residual block, which is mapped by the residual mapping and identity mapping [46]: y = F x , W i + x , (6)where x is the input feature information, y is the output feature information, and F ( x , { W i } ) represents the residual mapping process that needs to be learned. Next, the LBP image corresponding to the input image passes through a 64-kernel convolution layer, which is superimposed with the output feature information y. The superimposed result will skip connect with the 64-channel feature matrix below, which solves the problem of increasing training loss due to degradation, and makes the network show better robustness while maintaining the gradient flow. FIGURE 5Open in figure viewerPowerPoint Structure of proposed network. The network is mainly composed of four structures: convolution layers, residual blocks, ReLU layers, and self-attention block. The number in the figure represents the number of channels in the feature map of each structure. The internal structure of the residual block and specific structure of the self-attention module are shown in Figures 6 and 3, respectively FIGURE 6Open in figure viewerPowerPoint Internal structure of the residual block The feature matrix alternately passes through the miniaturized block composed of the convolutional layer and ReLU activation layer. The scale will go through two downsampling processes, from 256 ×256 to 128 ×128, then to 64 ×64, whereas the number of channels expands from 64 to 256 to encode the global information of the image. The feature matrix of 128 ×128 will be used for skip connect with the 128 ×128 feature matrix below. In the process of image scale decrease, the LBP feature also decreases scale and superimposes with the feature matrix in real time. Then the matrix passes through four continuous residual block regions, where the content features and color information of the image are organized and constructed. To return to the initial image scale, the reconstructed matrix passes through an enlarged block similar to the reduced block structure. First, the scale is enlarged from 64 ×64 to 128 ×128, and the enlarged feature matrix will be superimposed the LBP image of the same scale, and then skip connect with the map above. The feature matrix is enlarged again to initial scale 256 ×256, which also goes through the process of LBP superposition and skip connection. Next, the feature matrix passes through two residual block regions that are consistent with the above, so that the performance will not be affected as the network depth increases. After the next 3-kernel convolution layers, the number of channels is restored from 64 to 3. At this time we get the color-corrected image, but there are certain color deviation and weak visible stripe defects in the image, so we build the detail restoration module to fill in the details of the image. The input of the detail repair module is composed of the original image and resulting color correction image. By learning the residual information between the two images, continuous image information of the network output can be obtained. The superimposed image first passes through a 64-kernel convolution layer to convert the channel number to 64, and then overlays with LBP features that change the number of channels at the same time. Next, through the eight continuous residual block region, the detailed information of the image will be collected via superposition. Thus, more accurate details can be generated from the regional information near the defect point. After obtaining the detailed information, we put the obtained detail restoration matrix into the 256-kernel convolutional layer, and then perform the last superposition process with the LBP feature. We use the 3-kernel convolutional layer to convert the feature matrix into a detailed supplement image to repair the defect region of the correct color image. To further enrich the enhanced underwater image details, we also add a self-attention module before the detailed supplement image output. After testing, we found that this module can make the detailed image have more accurate environmental colors. Finally, we superimposed the color correction and detailed supplement image to obtain an outstanding underwater enhanced image. 3.4 Loss function The difference between the predicted images and actual effects during training is monitored to make the model produce more visual images. First, the common mean square error (MSE) is used to calculate the loss value of the model prediction result and ground truth as follows [47]: L m s e = E | | I g − I p | | 2 , (7)where I p represents the model prediction result graph, and I g represents the ground truth corresponding to the raw underwater image, L m s e is the mean value of the second-order norm of the feature map difference between the enhanced image and the ground truth. Although the model optimized using the MSE loss value can obtain a high peak signal-to-noise ratio (PSNR), MSE loss optimization results alone tend to be too smooth and lack high-frequency image details. Therefore, we added the loss L 1 that is closer to the pixel-wise calculation, and combined L m s e and L 1 to get the final loss result L f i n a l . The formula is as follows: L 1 = E | | I g − I p | | 1 , (8) L