Review of underwater image restoration algorithms

IET Image ProcessingVolume 13, Issue 10 p. 1587-1596 Review ArticleFree Access Review of underwater image restoration algorithms Jarina Raihan A, Corresponding Author Jarina Raihan A jari3010@yahoo.in Faculty of Integrated Technologies, Universiti Brunei Darussalam, Brunei DarussalamSearch for more papers by this authorPg Emeroylariffion Abas, Pg Emeroylariffion Abas orcid.org/0000-0002-7006-3838 Faculty of Integrated Technologies, Universiti Brunei Darussalam, Brunei DarussalamSearch for more papers by this authorLiyanage C. De Silva, Liyanage C. De Silva Faculty of Integrated Technologies, Universiti Brunei Darussalam, Brunei DarussalamSearch for more papers by this author Jarina Raihan A, Corresponding Author Jarina Raihan A jari3010@yahoo.in Faculty of Integrated Technologies, Universiti Brunei Darussalam, Brunei DarussalamSearch for more papers by this authorPg Emeroylariffion Abas, Pg Emeroylariffion Abas orcid.org/0000-0002-7006-3838 Faculty of Integrated Technologies, Universiti Brunei Darussalam, Brunei DarussalamSearch for more papers by this authorLiyanage C. De Silva, Liyanage C. De Silva Faculty of Integrated Technologies, Universiti Brunei Darussalam, Brunei DarussalamSearch for more papers by this author First published: 15 July 2019 https://doi.org/10.1049/iet-ipr.2019.0117Citations: 9AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Underwater images are susceptible to various distortions compared to images taken on land, due to the nature of the water environment. These images often suffer from diffraction, polarisation, absorption, scattering, colour loss and attenuation of light. Each part of the ocean will have its own sources of distortions, due to flickers caused by direct sunlight, marine snow, the fluorescence of biological objects, the presence of macroscopical organisms, loss of stability in divers, loss of light, artificial lighting and floating dust particles present in the water. There are numerous techniques and algorithms that may be used to restore these underwater images. This study reviews different algorithms and methods, developed in the past two decades, to give clearer ideas on the techniques present in the image restoration process, specifically for underwater images. 1 Introduction Underwater images are in many ways different from the images that are taken on air. The images taken underwater suffer from scattering, diffraction and colour absorption, as well as many other distortions. Sea water may be divided into three parts: the Euphotic, Disphotic and Aphotic zones, depending on its depth and light penetration levels. The Euphotic zone is the upper part of the ocean that receives bright and clear sunlight. This zone may extend to a depth of 80 m in clear tropical waters, but near the poles, it may just be 10 m deep due to sunlight not penetrating much. Beneath this zone is the Disphotic zone, which may extend to 800 m in clear water. The lowest layer beneath the Disphotic zone is the Aphotic zone where no sunlight enters. Images taken in the Euphotic zone are affected by flickers due to direct sunlight on suspended particles and absorption by floating microorganisms. In the Disphotic zone, backscattering, Rayleigh scattering and diffraction due to marine larval ecology are possible whilst in the deep Aphotic zone, loss of colours is a common issue along with strong visual cues from benthic features. Fig. 1 shows the different distortions in underwater images [1]. When a camera focuses on a target, its lens focuses on the target texture and sends out veiling light, which may be disturbed by particles and marine snow, as well as experiencing Rayleigh, Mie, Back, forward and multiple scatterings. Swimming macroscopical particles also lead to diffraction of the veiling light. Artificial light, normally used for taking images in dark underwater areas also undergoes Rayleigh and Mie scattering, as well as producing noise. Rayleigh scattering results in hazy images whilst Mie scattering results in blurry, murky and faded images. Fig. 1Open in figure viewerPowerPoint Types of distortions in underwater images, as of Rafael et al. [1] Differing colour absorption poses another threat to underwater images. The main reason behind this is that each colour has its own wavelength and frequency. Red light loses its strength after 5 m deep; which is why red fishes appear nearly black at 20 m. The green travels up to 30 m whilst blue light travels >50 m. Fig. 2 shows the light absorptions in water and their penetrating levels. In deep water, different penetrating levels of different colours result in underwater images to have a bluish green tint. Fig. 2Open in figure viewerPowerPoint Light absorption in water and penetrating levels of different colours, as of Chiang and Chen [2] Apart from these, underwater images also suffer from other problems such as focus, magnification and motion. Underwater, it is very difficult to point the lens exactly at the centre of focus of the target. Magnification is also a problem because the target is always 25% magnified when seen through the perspective of the snorkelers' goggles. This paper reviews some of the algorithms from the past two decades, which were developed for the restoration of underwater images. 2 Image restoration algorithms Distortions in the underwater environment lead to the development of restoration methods to allow recovery of images taken underwater. The restoration process involves the removal of blur and noise from an image. Blurring is the overall bandwidth reduction, which leads to an imperfect image formation model. On the other hand, noise is due to scattering effects and sometimes from the recording medium. The restoration process is based on a general Image Formation Model (IFM) (1)where I(x) is the observed intensity at pixel x. J(x), B(x) and t(x) are scene radiance, background light and transmission map at pixel x, respectively. Lu et al. [3] review a total of 34 papers from the literature on restoration methods for underwater images, by broadly classifying them into two approaches: hardware and software-based approaches. In contrast, three approaches have been reviewed in this paper: hardware, software and network-based approaches, and as such, provide a more comprehensive review on restoration methods for underwater images. Tables 1–3 show summaries of methods under hardware based, software based and network-based approaches, respectively. Table 1. Classification of the hardware-based approach Methods Details polarisers Schechner and Karpel [4] inversion of image formation model Treibitz and Schechner [5] visibility recovery approach ROVs Zhishen et al. [6] ROV based image processing range-gated imaging Tan et al. [7] reflected image temporal profile (RITP) + contrast limited adaptive histogram equalisation (CLAHE) Li et al. [8] speckle noise suppression stereo imaging Roser et al. [9] Bayesian approach Table 2. Classification of the network-based approach Methods Details CNN Anwar et al., [10] CNN Hu et al. [11] CNN with estimation network Shin et al. [12] ambient light and transmission estimation network Lu et al. [13] DCP with joint restoration filter Wang et al. [14] colour correction and haze removal GAN Fabbri et al. [15] GAN Li et al. [16] GAN with colour correction Table 3. Classification of the software-based approach Methods Details restoration filters Wang and Diao [17] distance-dependent formation algorithm Wang et al. [18] Weiner filtering with a knife-edge function Lu et al. [19] fast weighted guided normalised domain filtering algorithm Sahu et al. [20] un-sharp masking Wang et al. [21] Wiener filter with multilayer transfer model Trucco and Olmos-Antillon [22] self-tuning restoration filter Bazeille et al. [23] homomorphic filter colour correction Shamsuddin et al. [24] colour diminishment and stretching Kaeli and co-authors [25] colour correction techniques using information from sensor modules Iqbal et al., [26] equalisation of RGB colour space Petit et al. [27] colour dynamic compression using quaternions Iqbal et al. [28] contrast stretching in RGB colour space, saturation and intensity stretching in HSI colour space Gibson [29] contrast enhancement and turbulence mitigation Arnold-Bos et al. [30] adaptive smoothing Arnold-Bos et al. [31] Weiner filtering Chambah et al. [32] automatic colour equalisation Fu et al. [33] retinex based approach fusion Ancuti and Ancuti [34] multi-scale fusion Ancuti et al. [35] fusion principle Khan [36] fusion principle with wavelet transform Borker and Bonde [37] homomorphic filtering and adaptive histogram equalisation dark channel prior Xie et al. [38] dark channel prior to underwater reflection models Wang et al. [39] DCP based on blue-green channels Gupta and Faruq [40] DCP with FFT Galdran et al. [41] DCP with inverted red, green and blue channels Wen et al. [42] DCP with blue and green channels Drews et al. [43] DCP with blue and green channels He et al. [44] DCP with a median filter Yang et al. [45] DCP with soft matting Carlevaris-Bianco et al. [46] DCP with MIP Liu and Meng [47] DCP with filtering light and wavelength compensation Yang et al. [48] light reflection decomposition Wu et al. [49] underwater colour correction using depth compensation with non-local image de-hazing algorithm Li et al. [50] minimum information loss principle Emberton et al. [51] hierarchical rank-based approach Chen et al. [52] region-varied changes in the ambient light are taken into consideration for de-hazing and colour compensation Chiang and Chen [2] compensation for the artificial light source, wavelength compensation histogram equalisation Wong et al. [53] AGW and DHE Gao et al. [54] bright channel prior Hitam et al. [55] adaptive histogram equalisation Deperlioglu et al. [56] histogram equalisation in HSV space 2.1 Hardware-based approaches As its name implies, hardware-based methods use hardware components to directly restore underwater images. This includes the use of sensors, lasers, polarisers, polaricams, aqua tripods, Remotely Operated Vehicles (ROVs), laser-based methods and stereo imaging. Polarisation process can reduce backscattering to a certain extent and may be done, either by using polarisation cameras or by using a polarised light source to take pictures. To eliminate backscattering, laser imaging system uses a camera that can precisely close the flash gate at a specific point in time. Waterproof sensors may be used to sense macroparticles, marine snow, swimming biological organisms to allow rejection of reflections due to the particles. On the other hand, aqua tripods are placed on the seafloor to capture images more effectively. 2.1.1 Polarisers Schechner and Karpel [4] propose an image inversion model, based on images taken at different angles through a polariser. After analysis of the physical effects of visibility degradation, it has been concluded that the main degradation from underwater images is due to partial polarisation of light. To counter this, polarisers at different polarising angles are used to take pictures of the targets, which are then processed using an algorithm to invert the image formation process to improve visibility of the images. Dataset, from multiple images from the polarisers at different angles and orientations, is used to derive a distance map of the scene. Noise sensitivity of the recovered images is also analysed Two images are acquired, 26 m below the surface of the Red sea at two different polarisation angles using an Aqua-Polaricam; with the image in which only the water background is seen selected for further processing. The method can enhance stability, recover veiling light as well as derive a depth map of the image. Noise sensitivity problem of the images is also addressed by the authors. Fig. 3 shows the resultant image using the algorithm of an image taken in the Red sea. Fig. 3Open in figure viewerPowerPoint Comparison between the raw image (L) and the colour corrected image (R), as of Karpel et al. [4] Non-scanning recovery method is discussed by Treibitz and Schechner [5]. Using images with artificial illumination to study the 3D scene structure, the proposed method uses simple hardware and polychromatic polarised illumination. The polariser gives two frames of the scene, which follows a recovery algorithm for the acquisition. However, subsequent analysis on limitations and noise sensitivity shows that the method is limited within a certain effective range due to image noise. 2.1.2 Remotely operated vehicles ROV; a system, which is used for underwater detection and measurement, has proposed by Zischen et al. [6]. The ROV based system consists of artificial lighting, video cameras and thrusters, with the system kept at different sea conditions to obtain samples of the sea conditions. Radiance of light that is reflected is simulated using a computer, before using the Weiner filter on the images based on the observations of the sea conditions, to restore the blurred images. By using the method with proper parameters, restoration of blurry images in scattering seawater is shown to be practicable. 2.1.3 Range gated imaging Range gated imaging is applicable when the camera is adjacent to the light source with the target behind the scattering medium. With this image processing method, the camera is designed to only open its gate at a preconfigured short period of time, to eliminate backscattering of light. When the light source sends light pulse onto the object, the camera closes its gate as the pulse travels. The gate is only opened for a short period of time as the pulse returns after hitting the object and then, closed immediately thereafter. Tan et al. [7] propose a two-step design system: with hardware upgrades and system optimisation methods, for an underwater robotic vehicle involved in sub-sea inspections and repair missions, to improve underwater visibility. The hardware upgrades involve an advanced range-gated imaging system called Tail-gating, which uses the principle of delaying the camera gating process towards the tail of the RITP. CLAHE is then used as system optimisation, to further enhance the range-gated images. This method is similar to that proposed by Hitam et al. [55]. Li et al. [8] propose variations of the range gated imaging method, by using an imaging system which consists of a pulsed laser system, control and synchronous logics, and a high-speed gated camera. As a highly coherent light source is used, speckle noises resulting from the interference of scattered light rays, are expected in the images obtained from the system. To suppress these speckle noises, the multidirectional morphological filtering algorithm is used. The filtering algorithm has the characteristics of morphology's multi-resolution analysis and fast-computing, which includes square filtering and median filtering. Input images are sent to the square filtering window, with omnidirectional structuring element sets chosen, before passing it on to the median filter. Sub-images are then combined to form output image, which not only suppressed the speckle noise but also preserved feature details effectively. 2.1.4 Stereo imaging This type of imaging process involves the use of two cameras to take pictures of the same target at different views, to recover depth estimation information of the image. Roser et al. [9] propose a stereo-imaging method for autonomous underwater vehicles (AUVs) to recover underwater images in naturally lit turbid coastal environments, by estimating visibility coefficients. The proposed method is designed using real-time algorithms, consisting of two enhancement methods. The first image enhancement step attempts to recover important scene content and image structures in areas with bad visibility. For stereo retrieval of data, a Bayesian approach is used to create a prior on disparities from robustly matched support points. This prior yields a coarse 3D map of the scene. Using the physical light attenuation model and utilising the coarse 3D map, visibility coefficient, as well as ambient illumination, are estimated. To achieve a dense and edge-preserving depth map, an image matting approach is performed, which refines the coarse disparity map and extrapolates information to unmatchable areas based on observed intensity values, to form a ‘pseudo-3D map’. This image is then passed to the second enhancement stage where disparity gaps are filled using a Random Sample Consensus (RANSAC) process. These steps are repeated to create an ‘improved stereo’ map, which is further refined by utilising a discontinuity, by using preserving gap interpolation method, and by image inpainting for large areas with missing 3D information. The final map is then used with the gap interpolated map to recover scene radiances, as shown in Fig. 4. Fig. 4Open in figure viewerPowerPoint Comparisons between the (a) Input image, and output images using, (b) Method by Roser et al. [9], (c) Histogram equalisation method, (d) Adaptive histogram equalisation method as of Roser et al. [9] 2.2 Software-based approaches Software-based approaches use efficient algorithms to recover underwater images. The major classifications of the software approach are filters, DCP, convolutional neural networks (CNNs), colour correction, fusion, light and wavelength compensation, Histogram equalisation and so on. Normally, these techniques are based on assumptions on different coefficients which may sometimes be vague or take up presumed values. Compared to hardware-based approach, software-based approach scores well; giving less computational time, better designs, easy modulations and fewer investment costs. 2.2.1 Dark channel prior He et al. [44] introduce Dark Channel Prior (DCP). Although originally proposed for haze removal, it is the most common method used for underwater image enhancement process. The method is based on the observation that, in an image, there will be at least one channel with low intensity. This low-intensity channel may be used for depth mapping and creating transmission map. The three main reasons for the low-value pixels or dark pixels are shadows that fall on the image, colourful objects and dark objects. The next important assumptions used is that the dark pixels in an image are regarded as those close to the camera as they get less brightening effect, whereas the bright pixels are regarded far from the camera. However, this assumption fails in the presence of an artificial lighting source. Fig. 5 shows the picture taken with the artificial source with its foreground appearing brighter than its background. DCP is also not useful for deeper depths underwater. The reason is that red light has shorter penetration underwater, so while the darkest of the channels is considered for depth and transmission mapping, red channel always ends up being considered with the lowest intensity which gives erroneous results. Fig. 5Open in figure viewerPowerPoint Image with artificial light as of Chiang and Chen [2] As DCP experiences problems in processing underwater images, variations of the methods are subsequently proposed in the literature. DCP (GB) is proposed by Drews et al. [43], Wen et al. [42]; considering the green and blue channels only whilst neglecting the red channel. Drews et al. [43] estimate background light using (2)The red channel is suppressed with the assumption Irdark(x) and with the image restored using the equation. Wen et al. [42] propose suppressing the red channel by considering the prior p(x), using (3)Fig. 6 shows the results of image restoration using the method. This method is based on the fact that underwater images are predominantly bluish-green in colour and these two lights form the basis for underwater visual information. However, this method is found erroneous when the results are analysed based on red coloured target images. Fig. 6Open in figure viewerPowerPoint Comparisons between (a) original image, and output image from, (b) Fattal's result [57], (c) Carlevaris's-Bianco [46] result, (d) Wen's result, as of Wen et al. [42] Wang et al. [39] analyse images based on the blue-green channels, using a two-stage method. First, backscattering is removed using a binary quadratic function, before blue-green DCP is used to eliminate forward scattering and non-uniform lighting. Gupta and Faruq [40] formulate dark channel based method with fast Fourier transform (FFT) enhancement. Transmission map is obtained from residual energy ratios of the wavelength of each attenuating light, with morphological operations used for contrast adjustments. FFT is then used to recover the scene radiance. Fig. 7 shows the results acquired by Gupta and Faruq [40]. Fig. 7Open in figure viewerPowerPoint Comparisons between (a) Original Image, and output image using, (b) Quaternions method, (c) Dark Channel Method based on FFT, as of Gupta and Faruq [40] Other variation of DCP method is DCP (R′GB), which is proposed by Galdran et al. [41] that dealt with inverted red channels as well as the green and blue channels. This method rectifies the shortcomings of the DCP method for deeper depth; whereby the red channel always ends up being the darkest channel for the depth of >80 m. DCP (R′GB) effectively restores the red channel as well as colours with shorter wavelengths (4)where p is the prior for Ic and Ir(x) is the value of pixel x of the red channel. Values of pixels from the red channel are chosen from the minimum values of pixel x belonging to the 10% category of the prior among Ir(x). Xie et al. [38] use DCP method to find the darkest channel and then, underwater reflection model for restoration and enhancement of the image. Liu and Meng [47] put forward the method called the Dark Channel Prior (DCP), with the assumptions that in water free images, there is at least one channel which has very low intensity. By this way, the effect of water can be removed and the clarity in the images can be achieved. Fig. 8 shows the result of the algorithm. Fig. 8Open in figure viewerPowerPoint Comparisons between (a) Image before processing, (b) Image after processing with DCP, as of Liu and Meng [47] Yang et al. [45] propose the DCP method, but instead of using soft matting procedure, the median filter is used for restoration. Moreover, the authors also propose a colour correction model which enhances the contrast of the image. Using this method, execution time and computing resources can be reduced. 2.2.2 Restoration filters Filters are commonly used to remove blurriness in an image. Wang and Diao [17] use a distant dependant formation algorithm to restore the images; then a homomorphic filter is used to get rid of the non-linear disturbances in the distance-dependent model. The restored underwater image is then obtained by solving a Poisson equation that is derived based on the similar distance of neighbourhood pixels. Lu et al. [19] develop a fast weighted guided normalised convolution domain filtering algorithm for restoring underwater images. The algorithm also compensates for light attenuation along the propagation path. Sahu et al. [20] propose a method for enhancement based on un-sharp masking. Un-sharp masking is subtracting an image from its blurred version, which to clearly show the details in an image. The un-sharp masking is used to normalise the colour stretched image, and it is filtered to detect the presence of edges. The un-sharp mask is formed with a sharpened image, and the normalised image is used to increase the contrast and enhance the image. Wang et al. [21] propose an approach to formulate a model, which includes both Point Spread Function (PSF) and backscattered noise; with image restoration parameters estimated from in situ measurements of the background. Weiner filter is then applied for restoration purpose. A self-tuning image restoration filter based on a simplified version of Jaffe-McGlamery image formation model is proposed by Trucco and Olmos-Antillon [22]. The values for filter parameters are estimated automatically based on a global contrast measure. Fig. 9 shows the results of the original image processed with the self-tuning filter. Fig. 9Open in figure viewerPowerPoint Original image (T) and restored image with the self-tuning filter (B) as of Trucco and Olmos-Antillon [22] Wang et al. [18] develop a restoration method by designing the optimal window size and calculating motion blur parameters. Knife-edge function is used to determine the system degradation function. Finally, the derived image is passed through a Weiner filter to obtain the restored image. Bazeille et al. [23] design an algorithm by working in the YCbCr space. Initially, sampling is performed to eliminate the Moire effect. After resizing to a chosen value, RGB space is converted to YCbCr space, before it is passed through a homomorphic filter to eliminate non-uniform illumination. Wavelet denoising is then performed on the image. To improve image segmentation, the image is passed through an anisotropic filter, before final conversion back to RGB space from YCbCr space; to obtain the restored image. 2.2.3 Colour correction To correct the colours in RGB space, algorithms that evaluate every pixel based on colour absorption are developed. Singh and co-authors [25] propose a method by estimating attenuation coefficients and strobe beam pattern, using overlapped underwater colour images, with the help of a Doppler velocity log. These inputs are then used to colour correct the images. Iqbal et al. [26] design an algorithm based on colour balancing and colour correction in RGB and HSI colour space. As a first step, the colour cast is reduced by equalising the colour values. Red intensity is then increased by stretching the histogram whilst blue intensity is reduced by diminishing the histogram. HSI model is finally used for colour correction. Petit et al. [27] propose an algorithm based on light attenuation inversion. Geometric transformations are computed using quaternions with reference axis taken as a hue vector characterising the watercolour. Using this method, the contrast is significantly improved. Iqbal et al. [28] perform a two-step approach: first, contrast stretching in RGB space, and second, saturation and intensity stretching in HSI. RGB stretching equalises overall colour contrast of the image, resulting in the improvement of the image. Gibson [29] develop a method for turbulence mitigation and contrast enhancement to eliminate noise, increase local contrast, mitigate marine snow, and increase the sharpness of the image. In the developed method, noise is first removed with the help of a spatial median operator of size 3 × 3. Adaptive Gray-level Grouping filter is used to enhance the image contrast. With the images arranged globally, the adaptive correlation filter is used to estimate its global motion, whilst the optical flow method is used to estimate local motion. Consequently, frame averaging is performed with the help of a temporal median filter. Finally, the deblurring method is used to obtain the desired contrast-enhanced output, as shown in Fig. 10. Fig. 10Open in figure viewerPowerPoint Comparisons between (a) Original image, (b) Output image as of Gibson [29] Arnold-Bos et al. [30] develop an underwater image denoising algorithm using Current preprocessing method, by concentrating on local contrast. The authors have also shown that additional use of adaptive smoothing helps in denoising the remaining sources of noise and improve edge detection of image. As a first step, contrast equalisation is performed on the input image; by taking a low-pass filtered version of the image and passing it through a Gaussian filter, to address the backscattering effect. However, contrast equalisation naturally increases the amount of noise in fewer contrast areas of the input image. To counter this, the local gain is calculated and then multiplied by the standard deviation of the noise, to reduce the effect of noise in selective areas. Adaptive smoothing is then performed to preserve the edges; with less filtering in the edges to preserved the edges. Finally, the image goes through anisotropic and wavelet filtering processes to obtain the output. The authors further extend their work to address forward scattering in [31]; using forward scattering deconvolution method together with Weiner filtering. Fig. 11 shows the original input image, image results using the proposed method with adaptive smoothing without forward scattering deconvolution and with forward scattering deconvolution. Fig. 11Open in figure viewerPowerPoint Comparisons between (a) Input image, and final image after adaptive smoothing, (b) Without, (c) With forward scattering deconvolution as of Arnold-Bos et al. [31] Chambah et al. [32] propose a colour correction method for underwater images which is applicable for live fish recognition. For unsupervised image enhancement, Automatic Colour Equalisation (ACE) method is used, which merges the Gray version of the image with White Patch equalisation mechanisms, whilst taking into account of the spatial distribution of colour information. Thi