Underwater image restoration: A state‐of‐the‐art review

IET Image ProcessingVolume 15, Issue 2 p. 269-285 REVIEWOpen Access Underwater image restoration: A state-of-the-art review Sheezan Fayaz, Department of Electronics and Instrumentation Technology, University of Kashmir, Hazratbal, Srinagar, IndiaSearch for more papers by this authorShabir A. Parah, Corresponding Author drshabirparah@gmail.com Department of Electronics and Instrumentation Technology, University of Kashmir, Hazratbal, Srinagar, India Correspondence Shabir A. Parah, Department of Electronics and IT, University of Kashmir, Hazratbal, Srinagar, India. Email: drshabirparah@gmail.comSearch for more papers by this authorG. J. Qureshi, Higher Education Department, Government of Jammu and Kashmir, Jammu and Kashmir, IndiaSearch for more papers by this authorVijaya Kumar, School of Computer Science and Engineering, University of New South Wales, Sydney, New South Wales, AustraliaSearch for more papers by this author Sheezan Fayaz, Department of Electronics and Instrumentation Technology, University of Kashmir, Hazratbal, Srinagar, IndiaSearch for more papers by this authorShabir A. Parah, Corresponding Author drshabirparah@gmail.com Department of Electronics and Instrumentation Technology, University of Kashmir, Hazratbal, Srinagar, India Correspondence Shabir A. Parah, Department of Electronics and IT, University of Kashmir, Hazratbal, Srinagar, India. Email: drshabirparah@gmail.comSearch for more papers by this authorG. J. Qureshi, Higher Education Department, Government of Jammu and Kashmir, Jammu and Kashmir, IndiaSearch for more papers by this authorVijaya Kumar, School of Computer Science and Engineering, University of New South Wales, Sydney, New South Wales, AustraliaSearch for more papers by this author First published: 14 December 2020 https://doi.org/10.1049/ipr2.12041Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Underwater imaging is one of the hot areas of research and is receiving considerable attention from the research community due to the challenges involved. Underwater images are prone to various distortions like poor contrast and colour deviation. Light scattering and light absorption in water medium are the main reasons for degradation in subaquatic images. Scattering of visible-light energy reduces the sharpness of image whereas varying degrees of light attenuation travelling in water results in the colour change. Restoration of distorted underwater images is an ill-posed and challenging problem. Various techniques and methodologies are being used to process and restore underwater images. In this study, we present a state-of-the-art review of various conventional and computer vision-based algorithms and techniques, developed so far, to present a clearer view of the methods used for underwater image restoration. We discuss various conditions for which the schemes have been developed as well as highlight the quality assessment methods used to evaluate their performance. We compare various state-of-art schemes based on various subjective and objective indices and discuss future research directions in the field of underwater image restoration. 1 INTRODUCTION In ocean engineering, obtaining undistorted images in the water medium is a major issue. Oceans have a large amount of means of energy, which plays an essential role in the prolongation of existence on earth, and aquatic survey involves high technological activities [1, 2]. Fog and haze removal are mandatory for applications that are instantaneous, for example, transportation that is video-directed [3-5], environment monitoring [6-8], for surveying instantaneous remote detection pictures [9-12], and the auto-pilot supported structures [13-15]. In addition to this, it is quite challenging to precisely recover remotely sensed objects covered by thick clouds and shadows [16]. Also, the processing of digital pictures is challenging and increasing speedily. Mist in the picture seems to be assorted pels in remote detection [17]. Furthermore, blurriness leads to the deterioration of ocular information [18]. The quality of the underwater image is essential for analysing aquatic life, studying biological or geological abode [19]. A lot of focus has been attracted by ocular technology as it contains ample information [20]. Chemical composition and physical characteristics of the environment underwater immensely affect the quality of pictures captured in such an environment, leading to problems that are not encountered in imaging above water [21]. Moreover, the density of the water is about 800 times greater than the atmosphere above water. Thus, the portion of visible light which falls on water partly bounces back and partly goes into the water [22]. The portion of energy going into the water begins to decrease as it propagates in the water. Molecules of water absorb the portion of the energy of light and that is why the subaquatic images appear more unlit with an increase in depth. Particles being suspended in water reduce maximum energy and deviates light energy from its path before it reaches the camera, resulting in a blur, reduced contrast, and mist [23]. Also, because of absorption, scene radiance reduces with the increase in distance of the desired object from the camera [24]. Depending upon different wavelengths of light, colours drop gradually as the visible-light energy propagates deeper. At first, the longest wavelength, that is, red drops at 3 m of depth, and then the orange wavelength is lost at the depth of 5 m. After that, the yellow wavelength disappears at about 10 m, and eventually, the green and purple wavelengths disappear [25]. Blue colour travels the longest in the water due to its shortest wavelength. This is what makes the underwater images dominated only by blue colour [26]. Figure 1 shows how colour drops off as light travels deeper underwater. Also, molecules of water reduce some percent of visible-light energy which causes haziness of an image [27]. As visible-light coming from the image travels toward the aperture of the camera, part of this visible-light energy encounters the particles suspended in water. This results in the sucking up and scattering of light energy by these particles as illustrated by Figure 1. Artificial illumination is frequently used but it is also influenced by scattering and attenuation [28]. Simultaneously, uneven illumination is caused, leading to spots that are bright at the centre of the subaquatic image [29]. FIGURE 1Open in figure viewerPowerPoint Diagram of subaquatic ocular imaging Three types of light radiations are received by the camera: Energy bounced back from the scene directly (direct transmission), energy that comes across tiny particles and gets scattered before reaching the aperture of image-capturing device (forward-scattering), and energy from atmospheric light that is bounced by the particles present in water (background-scattering) [30]. An undersea image is shown as a linear combination of direct-transmission, forward-scattering, and backscattering components [31-33]: E t l , m = E d l , m + E f l , m + E b l , m (1)where (l,m) denotes the coordinates of picture elements; Et (l,m) denotes the overall energy coming toward the aperture of the camera. As the distance of the undersea scene and camera aperture is relatively small, the forward-scattered energy can be neglected and just the rest of the two components [34, 19] are considered. Given all intricacies involved regarding the formation of the underwater image and the various underwater distortions that creep into the process, the underwater image restoration is a tedious task. In this study, we mainly focus on image restoration algorithms. Multiple contributions to the study of various undersea image restoration algorithms are made and are presented as follows. We divide underwater image restoration techniques into two main categories: Hardware-driven techniques and software-driven techniques (prior-driven image restoration methods and network-based image restoration methods). We show the comparisons based on some representative techniques from prior-driven, which provides an assessment based on different quality metrics. We also discuss the issues in different underwater image restoration algorithms leading to new research directions. The remaining portion of the study is as follows. In Section 2, we discuss the concept of image restoration. Section 3 discusses various hardware-based restoration methods. Section 4 discusses various software-based restoration methods. Ed(l,m) denotes direct-scattering component, Eb(l,m) denotes backscattering component, and Ef denotes forward-scattering component. Section 5 shows a comparison of various subaquatic picture-recovering techniques through experiments. Section 6 discusses advantages and disadvantages of current picture-restoration techniques and Section 7 concludes the discussion and puts forward the future directions. 2 IMAGE RESTORATION The methods of image restoration are based on degradation principle unlike image-enhancement methods. The methods of image enhancement do not take into consideration the relationship between the depth and deterioration of images. Such kinds of techniques enhance the visual effects but do not reflect the true colour of an image correctly. The enhancement technique corrects the sharpness and the visual quality of the picture but does not take away the full haze [6]. Some image-enhancement methods used for enhancing underwater images are one-scan shadow compensation and visual enhancement of colour images [35], underwater image enhancement via extended multiscale retinex [36], fusion-based underwater image enhancement [37], automatic system for improving colour and contrast using adaptive histogram [38], subaquatic image enhancement using integrated-colour model [39] and subaquatic image enhancement using hue-preserving approach [40], Rayleigh distribution (RD)-based enhancement [41] and relative-global-histogram-stretching (RGHS) [42]. The above-mentioned techniques over-enhance pictures because these methods do not depend on prior data of subaquatic conditions [43]. The representation popularly used for misty picture in computer vision is [44-47]: I ( x ) = J x t x + A 1 − t x (2) where I denotes the input intensity at point x, J denotes scene-radiance at point x, A denotes global ambient-light energy and t denotes medium transmission map (TM) describing part of the energy that reaches the camera aperture without getting scattered. The above-mentioned model is popularly called the image formation model (IFM) and is given by Duntley et al. [48]. We use Duntley's representation for subaquatic image recovery. In the above equation, both terms give the theoretical idea of images with blur and haze [49-52]. The main aim of dehazing is to compute the value of J which can be calculated after calculating the two important parameters, that is, the global ambient light and TP. Therefore, these unknowns need to be estimated for the restoration of an image captured underwater. Based on the available studies, the methods of image restoration are classified into two broad areas: Hardware-driven and software-driven approaches (prior-driven and network-driven techniques). 3 HARDWARE-BASED APPROACHES These approaches use hardware equipment to recover an underwater image. Some hardware approaches include components like sensors, lasers, polarisers, remotely controlled automobiles, stereo imaging, aqua-tripods, and methods based on the laser. Polarisers can diminish the effect of backscattering to some degree and can be implemented by utilising polarisation cameras or utilising a light energy source which is polarised to capture images [53]. Not to have the effect of backscattering, laser imaging makes use of an image-capturing system that can shut the gate of flash at particular instant. Macro-particles, subaquatic snow, subaquatic organisms can be detected by waterproof sensors to eliminate reflections due to the suspended particles. Aqua-tripod is positioned on the surface of the sea to click images properly. Some of the main hardware methods are described as follows: 3.1 Remotely controlled automobiles It is used for the undersea survey and has been presented by Zischen et al. [54]. It has an artificial lighting system, thrusters, and video cameras. This equipment is positioned at different places, facing different water environments to capture samples of these environments. A computer simulates the reflected visible-light energy. Before the above step, the Weiner filtre is applied to the images which takes into consideration the analysis of subaquatic conditions to recover the blurry pictures. When the technique is implemented with an appropriate framework, recovering distorted images is expected to be practicable. 3.2 Polarisers Schechner and Karpel [55] had put forward the inversion model of the image which is based on pictures captured at various angles using a polariser. The maximum distortion in subaquatic images is because of light being polarised partially. To address this issue, polarisers are used at multiple angles of polarisation to capture images of the targeted object. These pictures are then processed by utilising a technique to upturn the image-formation process to enhance visibility. A scene-distance map is calculated using a collection of images from the polarisers. The recovery method without scanning is given by Schechner and Treibitz [56]. Utilising artificial illumination to analyse the structure of 3-D scenes, this approach uses some hardware and polarised multi-coloured lighting. Polariser outputs scene with two frames, following a restoration algorithm for acquisition [53]. Further understanding leads to the conclusion that the particular approach is restricted within some range because of noise. 3.3 Range-gated imaging This approach employs the light source in addition to the camera and the desired object lies behind the complex environment. It is the type of image processing in which the camera opens the gate precisely at a predefined short instant, to reduce the effect of backscattering. After projecting light toward the targeted object by the illuminating source, the camera shuts the gate as light travels. When light energy is bounced back from the desired object, the gate of the camera opens. Then, the gate shuts immediately after that. Tan et al. [57] found a design comprising of two steps: With upgraded hardware and optimisation techniques, for subaquatic robotic automobile used in undersea surveying and repairing, to enhance undersea visibility [53]. Upgraded hardware consists of an efficient range-gating for imaging known as tailgating. System optimisation uses CLAHE, to make more improvements in the pictures. This approach is the same as the one presented by Hitam et al. [58]. Li et al. [59] put forward different range-gated imaging techniques, by utilising a pulsed laser for picturing and using fast-gated camera [53]. 3.4 Stereo imaging This method uses a bi-camera configuration to click pictures of the same spot at various angles to obtain a distance map of the object. Roser et al. [60] uses a technique of stereo imaging for automatic undersea automobiles to restore subaquatic pictures in naturally illuminated murky mediums by predicting coefficients of visibility. This technique involves two methods to improve images. The initial step involves restoring scenes in regions with poor visibility. For the recovery of data, the Bayesian approach is employed to produce a prior. The prior then produces a rough 3-D scene map. The physical-light absorption model and rough 3-D scene map are employed to predict ambient lighting and visibility coefficient. To obtain a depth map that preserves edges, image matting is carried out refining the coarse map and deriving information that depends on appeared intensities, to obtain the so-called 3-D map [53]. The resulting data is then processed using another enhancement step where spaces are filled. The two-step technique is then performed again to generate enhanced stereo-map. This map is then improved using the preserving gap interpolation technique. Eventually, the resulting map is then used in combination with the map with interpolated gaps to restore an image. 4 SOFTWARE-BASED APPROACHES These algorithms employ effective methods to restore subaquatic images. Broad classification of software-driven techniques used for subaquatic image recovery are prior-based methods and network-based methods. Software-based approaches perform well as compared to hardware-based approaches. Software approaches consume less time in computation, have feasible designs, and are economical (cost-effective). 4.1 Prior-based methods Absorption of light energy and its scattering due to suspended particles mainly contribute to the degradation of underwater images. Considering discriminatory light attenuation and hazy effect, various prior-driven methods are used for undersea image restoration [21]. These are: Dark channel prior (DCP) [61, 62]; underwater DCP (UDCP) and transmission estimation in underwater single images (TEoUI; UDCP-TEoUI) [63], UDCP [64]; maximum intensity prior (MIP) and initial results in underwater single image dehazing (IUID; MIP-IUID) [65]; red channel prior (RCP) [66]; blurriness prior (BP); underwater light attenuation prior (ULAP) [67]; low-complexity UDCP [68] and so forth. These priors first calculate atmospheric light and medium TM. Then the calculated values are substituted into the IFM representation used for image recovery [21]. Some prior data has been proposed in past years to compute the medium TM. These methods depend on prior information. The transmission reduces to zero gradually as the scene depth increases and the contribution of atmospheric light becomes significant. So, pixels in an image with maximum scene-depth are considered for computation of atmospheric-light energy. Selecting these pixels with accuracy, two rules are proposed. The pixel in an image with the highest brightness can be considered as atmospheric light is proposed by Chiang and Chen (2012). Galdran et al. and Li and Guo, in 2015, proposed that the intensity of reference pixels in the red channel is lower than B and G channels. However, the reference pixels are selected based on colour information by both the rules. Depending on colour information, the prior-based methods underrate the transmission of blue or green colour. 4.1.1 Various prior driven algorithms Dark channel prior (DCP) DCP is proffered by Kaiming He et al. [69]. It is a well-known dehazing technique used for haze removal of degraded outdoor images. As hazy images are similar to the images taken underwater, this technique is also used for subaquatic image dehazing. The said method uses the fact that in one colour channel of an image without haze, some of the pixels have minimum intensities. If DCP is directly used for dehazing of undersea images, background-light energy can be calculated by following the below steps: In dark channel of a picture, select topmost 0.1% of the brightest pixels and out of these pixels, choose the highest intensity pixels [68]. By substituting the brightest value of a pixel as atmospheric-light energy in Equation (2), the TM can be estimated from Equation (6). Calculating dark channel from Equation ( (2) and by applying minimum operators on both sides of Equation (2), denoting minimum value of scene radiance J as Jdark is shown as the below: J dark → 0 (3) J dark x = min Y e π x min C ∈ r , g , b J C ( Y ) (4) where C, ϵ, r, g, b and denote particular channels, π(x) denotes a local patch centred at x on an image. Using minimum operators on both sides of Equation (2) and normalising w.r.t. Ac we get: min Y ∫ π x min C I C y A C = t x ) min Y ∫ π x min C J C y A C + 1 − t x (5) where Ac denotes atmospheric light in a particular channel. Using Equations (3) and (4) in above, we get: t x = 1 + min Y ∫ π x min C I C y A C (6) Since t(x) is considered constant in the patch π(x), some block artefacts and halos are obtained. This is because the TM is not constant. Therefore, to refine the TM soft matting or filtering can be used [69, 68]. Chao and Wang [70] applied DCP directly to restore the underwater images and remove the haze present in them. However, the recovered images appear less natural as they suffer from additional colour deviation. Yang et al. [68] in 2011 presented an underwater image-restoration technique based on DCP to decrease computation and execution time. They used median filtering instead of soft matting for refinement of TM which contains the image-depth information. This technique also proposes a way out for the correction of colour to correct the brightness of recovered images. However, it is suitable to only colour-rich images and not for recovering undersea images with a dim scene or uneven colour spots [21]. Chen and Chiang [19] presented image dehazing and wavelength compensation to reduce the effect of artificial illumination, for compensation of three-colour channels with varying absorption characteristics, and dehaze the image [21]. Lu and Serikawa [71] used a combination of fast joint trigonometric filtering (JTF) and DCP, for the compensation of discrepancy in attenuation in the light propagation path. This filtering technique is used for medium-transmission refinement that is calculated by the conventional DCP for removing non-uniform colour-cast and correct contrast and preserving information in edges in images. Zhao et al. [72], assessed subaquatic ocular properties by analysing the colour of the background area in subaquatic or undersea images. The coefficients of attenuation of three colour channels namely, RGB are calculated from the relation between background light (BL) and ocular properties that are inherent. The conventional DCP algorithm was considered for estimation of the TM of the R channel, and after that TM of GB colour channels were estimated from the R channel by using exponential-dependence relation between attenuation coefficients. Peng et al. [73] selected top 0.1% of pixels with the highest intensity in dark channel and a calculated average of selected pixel intensities which was then considered as atmospheric light. DCP gets influenced by the attenuation of light energy, which is selective in conditions as in water, which resulted in the development of various DCP algorithms specific to underwater environments. Underwater DCP-driven image restoration: When light energy travels in water it undergoes attenuation. As we know, the wavelength of red colour is much larger than other colours comprising the visible-light spectrum, so red colour drops faster than the other colours in light energy. Therefore, R channel of subaquatic image is usually considered as dark channel. However, to nullify the effect of red colour, UDCP [63], by Drews et al., calculates dark channel of an underwater picture by considering blue and green channels only. To estimate rate of scattering, Wen et al. [74] proposed an optical model for undersea images which resembles UDCP. UDCP estimates the TM more accurately than traditional DCP, however, the images restored by UDCP are less satisfactory. The reason for the unsatisfactory results is that these techniques do not consider the imaging characteristics of R and GB wavelengths. Lu et al. [75] proposed that the blue channel can also have the lowest-pixel value occasionally in turbid water. The R channel among the RGB channels does not always have the minimum pixel value as in the case of turbid water. Thus, they proposed the method of using a dual dark channel comprising of B and R channels to calculate the rough TM and removed halos in the estimated medium transmission by utilising a weighted median filtre. Lu et al. [76] found the super-resolution of an underwater image by taking a scattered image with high resolution (HR) and descattered images. Post this, the rule of convex fusion is applied for restoring the final HR picture. Guojia Hou et al. [77] proposed a new curvature and total variation-based method to denoise and dehaze pictures simultaneously. Bright channel prior (BCP) The medium transmission is estimated via the bright channel. Colour channels are first separated into three channels as follows [78]: I r = J r t + A r 1 − t (7) 1 − I g = 1 − J g + ( 1 − A g 1 − t (8) 1 − I b = ( 1 − J b ) + 1 − A b 1 − t (9) where t denotes the medium transmission, Ir, Ig, and Ib denotes three colour channel pictures of the distorted undersea image, respectively. Jr, Jg, and Jb are the scene radiances of three colour channels [78]. Ar, Ag, and Ab are atmospheric lights of R, G, and B pictures of the deteriorated undersea image, respectively [78]. The images or scene radiances without degradation (Jr,1−Jg,1−Jb) are shown as Jn and the degraded image is shown as In. An denotes the atmospheric light in the degraded image: I n x = J n x t x + A n 1 − t x (10) Using three colour channels, Equation (10) is deformed as I n c A n c = t x J n c A n c + 1 − t x (11) c {rn, gn, bn} represents the three colour channels Bright channel is defined as J n b x = max Y ∫ Ω x max C ∫ r n , g n , b n J n c y (12) where Ω(x) is a local patch of an image. The bright channel intensity of undersea pictures with murky water areas and far away scenes are approximately 1 [78]: J n b → 1 (13) Taking maximum values on both sides of Equation (11), t(x) can be calculated using Equation (15): max Y ∫ Ω x max C ∫ r n , g n , b n J n c y = 1 (14) t x = 1 1 − A n c max Y ∫ Ω x max C ∫ r n , g n , b n I n c y − A n c 1 − A n c (15) Image with the highest colour difference is defined as follows: B g − s u b r x = 1 − max max C max x − C min x , 0 max C mid x − C min x , 0 (16)where Bg–subr is a colour image with the highest difference, x is the location of pixel, Cmax is the channel with the greatest intensity in three channels, Cmid has a medium intensity in three channels, and Cmin has a minimum intensity in three channels [76]. The transmission obtained is smaller than the actual transmission as in BCP, the bright channel of the image without degradation is assumed approximately equal to 1. Therefore, to increase the stability, the calculated bright channel of a picture is corrected by using the maximum colour difference image [78]. The equation used is l i g h t c o r r e c t x = λ ∗ l i g h t x + 1 − λ ∗ B g − s u b r x (17) where light correct(x) represents the corrected bright channel picture, l i g h t ( x ) = ma x y ∈ Ω ( x ) ( ma x c ∈ { r n , g n , b n } I n c ( Y ) ) represents the non-rectified bright channel picture of a deteriorated undersea picture, and λ is a proportional coefficient. To compute global atmospheric-light energy, a bright channel of a degraded picture is used. The variance image of the Grayscale version of the original image is produced. Then, 1% of pixels with the lowest intensity in the bright channel are picked. From these pixels, a pixel with the minimum value in the variance picture is chosen as atmospheric-light energy. Finally, image without haze is recovered using Equation (10) as all unknowns are calculated. The whole process is described in Figure 2. FIGURE 2Open in figure viewerPowerPoint Bright channel prior (BCP) algorithm flow Maximum intensity prior (MIP) Exploiting the strong variation in attenuation between GB and R channels of subaquatic pictures, Carlevaris-Bianco et al. [65] found a method wherein a great difference between the absorption of RGB wavelengths was discovered. Based on this prior data, computation of depth of the scene is carried out. The proposed algorithm is thus called MIP. The variation between two extreme values of R-channel intensity and BG-channel intensities is used for medium-transmission estimation. The experiment proved that MIP can compute rough distance maps of underwater pictures [21]. The proposed algorithm can be utilised for atmospheric-light estimation as well. Wen et al. [74] proposed the improved version of MIP (i.e. new optical model (NOM)) to compute the atmospheric light. The assumption used was that the intensity of the R wavelength is comparatively smaller than the intensity of the green-blue wavelengths in the area in the background. Zhao et al. [72] proposed a method for estimation of BL using MIP and DCP. Upper 0.1% of the highly bright pels present in dark channel were selected first, then pel with the highest variation between blue-green wavelengths or GR wavelengths was selected [21]. Li et al. [79] estimated the