UWGAN: UNDERWATER GAN FOR REAL-WORLD UNDERWATER COLOR RESTORATION AND DEHAZING
Authors: Nan Wang
Yabin Zhou
Fenglei Han
Lichao Wan
Haitao Zhu
Yaojing Zheng
Venue:  (2020)
Keywords: underwater image
image restoration
image enhancement
GAN
CNNs

Abstract: In real-world underwater environment, exploration of seabed resources, underwater archaeology, and underwater fishing rely on a variety of sensors, vision sensor is the most important one due to its high information content, non-intrusive, and passive nature. However, wavelength-dependent light attenuation and back-scattering result in color distortion and haze effect, which degrade the visibility of images. To address this problem, firstly, we proposed an unsupervised generative adversarial network (GAN) for generating realistic underwater images (color distortion and haze effect simulation) from in-air image and depth map pairs. Secondly, U-Net, which is trained efficiently using synthetic underwater dataset, is adopted for color restoration and de-hazing. Our model directly reconstructs underwater clear images using end-to-end autoencoder networks, while maintaining scene content structural similarity. The results obtained by our method were compared with existing methods qualitatively and quantitatively. Experimental results on open real-world underwater datasets demonstrate that the presented method performs well on different actual underwater scenes, and the processing speed can reach up to 125FPS on images running on one NVIDIA 1060 GPU.

Under review conference paper at ICLR 2020
UWGAN: UNDERWATER GAN FOR REAL-WORLD UNDERWATER COLOR RESTORATION AND DEHAZING
Anonymous authors
Paper under double-blind review
ABSTRACT
In real-world underwater environment, exploration of seabed resources, underwater archaeology, and underwater fishing rely variety of sensors, vision sensor is the most important one due to its high information content, non-intrusive, and passive nature. However, wavelength-deeennden light attenuation and back-scattering result in color distortion and haze effect, which degrade the visibility of images. To address this problem, firstly, we proposed an unsupervised generative adversarial network (GAN) for generating realistic underwater images (color distortion and haze effect) from in-air image and depth map pairs based improved underwater imaging model. Secondly, U-Net, which is trained efficiently using synthetic underwater dataset, is adopted for color restoration and dehazing. Our model directly reconstructs underwater clear images using end-to-end autoencoder networks, while maintaining scene content structural similarity. The results obtained by our method were compared with existing methods qualitatively and quantitatively. Experimental results obtained by the proposed model demonstrate well performance on open real-world underwater datasets, and the processing speed can reach up 125FPS running one NVIDIA 1060 GPU.
INTRODUCTION
In recent years, underwater vision plays an important role in lot of different applications. Therefore, underwater image processing has received extensive attention and research due the poor underwater imaging environment and image quality. The main reason is the scattering and attenuation of light, the scattering results haze effect, and the attenuation of light leads color cast.
So far many image enhancement algorithms have been proposed, such as white balance algorithm (Liu C (995),, gray world algorithm (Rizzi AJ, 2002). histogram equalization (Pizer S M. 1987) and fusion algorithm (Ancuti C, 2012),, however, these methods are not based on the underwater physical imaging model, it is challenging and ineffective to apply these algorithms to different underwater scenes directly.
Many underwater image enhancement algorithms based on imaging models have been proposed. For instance, He et al (He K, 2010) proposed a dark channel prior (Dark channel prior, DCP) dehazing algorithm based on many experiments. Chiang al (Chiang Y, (011) apply DCP model on underwater image dehazing problem. These traditional methods are not intelligent, it is very time-consuming to calculate the characteristics of the image.
In these years, the deep learning network developed rapidly, especially the convolutional neural network (CNN), which is used in image classification (Krizhevsky Al, 2012), object detection (Redmon J. 2016),, and motion recognition (Kuehne H. 2011), the performance much better than traditional methods. However, the current research on underwater image enhancement using CNN is limited due to lack of underwater datasets. It is difficult to obtain images without water in real-world underwater seenes. Therefore. using synthetic underwater datasets an important approach (Anwwr S) 20188; Ancuti C. 2016 Uplavikar P. 2019). Some model based on generative adversarial network (GAN (Goodfellow 11. 2014)) are used to generating realistic underwater images. For instance, CycleGAN (Zhu J Y, 2017) generates images through

---

Under review conference paper at ICLR 2020
style transfer. WaterGAN (Li
2017) takes in-air images, depth maps and noise vectors as input, followed by camera model, then output synthetic images. Based our experimental results, the image generated by WaterGAN suffers color noise and they differ a lot from real world underwater images. Therefore, to generate realistic underwater images with both color cast and haze effect, we improved the underwater imaging model, and proposed an unsupervised GAN based on this model to generate realistic underwater images from clear in-air images. Then, U-Net with different loss functions (Ronnebergerr O (015)) is trained to enhance underwater images through synthetic datasets. Finally, the performance of the proposed algorithm is validated on real underwater images as well as underwater target detection datasets for both low-level and high-level computer vision tasks. The experimental results show that the proposed method can recover the underwater image while maintaining structural similarities. Apart from this, the effects of different loss functions in U- net compared, the most suitable loss function for underwater image restoration is suggested based on the comparison (This part can be found in APPENDIX), which provides a new idea for underwater image enhancement.
2
OUR PROPOSED METHOD
To generate the realistic underwater images (color casts, low contrast and haze effect), we improved underwater imaging model, and proposed an underwater generative adversarial network (UWGAN), which takes in-air RGB-D images and sample set of underwater images of specific survey site as input to train a generative network adversarially. These synthetic underwater images, which were used to train a restoration network based on U-Net (Ronneberger O 2015) that can enhance underwater images in real-time.
2.1 IMPROVED UNDERWATER IMAGING MODEL
As is well known, simplified underwater imaging model is shown in Eq.
1(x) ](x) + A(1 - T(x)
Txx) = ((())(())
(Eq. 1)
where, 1(x) is the light intensity of each pixel x. (xx is the initial irradiance that not propagating through the water. T(x) is the transmission map of the scene. A is the atmospheric ambient light of the scene. B is attenuation coefficient of light of different wavelengths A, and d(x) is the range between the scene and the camera.
(a)
(b)
(c)
(d)
Figure 1. Synthetic underwater-styl images through Eq. 2. (a) are in-air sample images, (b)-(d) are synthetic underwater-style sample images of different water types.
2

---

Under review conference paper at ICLR 2020
We can generate underwater-style images using the in-air image and its depth map by Eq. D. which can well simulate color cast caused by light attenuation in water. However, it is difficult to simulate the haze effect caused by the scattering of water impurities. As shown in Figure 44 obvious haze effect can be observed on real underwater images. Inspired by related dehazing methods (AAncuti C, 2016),, we improved the second term in Eq. [1. The improved imaging model is shown in Eq. 2
1(x) = [())(() + (1
(x) -ad(x)
(Eq. 2)
where, AT(x) is ambient light based the light attenuation of different wavelength. is the scene scattering coefficient, which corresponds to the scattering coefficient in the atmospheric imaging model, and is set by default to corresponding to moderate and homogeneous haze effect. Three types of realistic underwater images were synthesized with color cast and haze effect are shown in Figure
2.2 UWGAN FOR GENERATING REALISTIC UNDERWATER IMAGES
Underwater-style images are generated based on Eq. whose parameters are estimated through adversarial
learning using GAN, as shown Figure 2
Generator
RGB Image
Imago
0/1
Forward Seatter-
Discriminator
TiX)
Depth Map
RGB Image
RCH Image (rei underwaze))
Attencation
Dogradod-++
RGB Image
RGBB Image (syminetic underwated)
(in-ak))
Figure 2: UnderwaterGAN architecture. UWGAN takes color image and its depth map as input, then it synthesizes underwater realistic images based underwater optical imaging model by learning parameters through generative adversarial training.
2.3 UNDERWATER IMAGE RESTORATION
U-NET
U-Net used for color restoration and haze removal of underwater images. detailed description of U-Net architecture proposed in the paper is shown in Figure 3. Firstly, a degraded underwater RGB image is resized to 256x256 and then fed into the encoder part of U-net. In the encoder, the image finally downsampled into 12x32x256-dimensee
latent vector through series of convolution and max-pooling operations. In each downsampling stage, 3x3 convolution with a stride followed by a rectified linear unit (ReLU) activation function are conducted twice, then 2x2 max pooling with a stride of is used. The number of feature maps are doubled at each stage. In the decoder part, upsampling is done from the latent high dimensional vector back to the original input size sequentially. After each upsampling operation, output tensor is concatenated to the corresponding symmetric layer in the encoder side, then followed by two consecutive convolution layers and rectified linear activation layer. The number of feature maps gradually reduced to three channels.
3

---

Under review conference paper at ICLR 2020
Ouia Rcorstvant Fangs
11A 175
Goaed
Figure 3: Proposed U-net Architecture for underwater image restoration and enhancement.
2.4 DATASET
The in-air datasets used are images of indoor scenes that has been labeled in the NYU Depth dataset VI (Silberman NJ 2011) and V2 (Silberman Nj 2012), which contain a total of 3733 RGB images and corresponding depth maps. The underwater dataset contains real-world underwater images collected from marine organisms' farms (including scallops, sea cucumbers, sea urchins, etc.), which can be roughly divided into two categories, one contains near-field green hued images (RealA), and the other contains blue-green hued images of far-field scenes (RcalB). We also use underwater open datasets (Li C. (019) (RealC) as testing sets, where RealA contains 2069 underwater images, RealB contains 2173 underwater images, and RealC contains 890 underwater images. Several typical images of the datasets are shown in Figure 4
(a)
(b)
(d)
(c)
Figure Typical images of datasets. (a)-(b) are color images and depth maps of NYU-Depth datasets, (c) are sample images of RealA dataset, (d) are sample images of RealB dataset, (e) are sample images of RealC datasct.
3 EXPERIMENTAL SETUU
The training settings of our proposed method are presented in details in this section. Our models trained in the computer with the following configurations: Intel i7 HQ 8700 processor, 16GB RAM, NVIDIA TITAN X 12GB graphics card.
Firstly, UWGAN is trained synthesize underwater-style images using the NYU-Depth Dataset, RealA and RealB datasets. Our model was trained for 30 epochs, using Adam optimizer with learning rate of 0.0001, and the momentum term was set to 0.5. The batch size was set to 64 with output images set to 256x256. Secondly, U-net is trained 32 and the output image size is 256x256. The learning rate is set to 0.0001 according to Adam optimizer, our model is trained for 200 epochs.
image enhancement network using synthetic pairs. The batch size was set to
RESULT AND DISCUSSION
4

---

Under review conference paper at ICLR 2020
In this section, we quantitatively and qualitatively compare our proposed method with several representative underwater image enhancement algorithms, including Unsupervised Color Correction Method (UCM) (Igbal K, 2010),, Histogram equalization (HE) (Hummel R. 1975). Multi-Scale Retinex with Color Restoration (MSRCR) (Rahman Z, I996), Fusion (Annctti C, 2012), Underwater Dark Channel Prior (UDCP) (Drews P. 2013), Image Blurriness and Light Absorption (IBLA) (Peng Y II (017)), Underwater Color Correction
using GAN (UGAN) (Fabbri C. 2018), VateerGAN--cooor-ccorettio (WaterGAN) (Li 2017).
(8)
(h)
(c)
(d)
(c)
(h)
Iapat
DCM
MSSRCR
LDCP
UGAN
Water GAN
Oars
Figure 5: Qualitative comparisons for samples from the real-world underwater image dataset RealC. (a)-() represent the samples selected from RealC.
We employ a non-reference metric, UIQM (Panetta K. 2015), for the quantitative assessment of underwater image quality on RealA, RealB, and RealC datasets as no ground truth scenes available as the reference for real-worke underwater images. Besides, we employ three full-reference metrics, namely MSE, PSNR (Hore A,, 2010), SSIM, for assessment image quality on synthetic datasets. To reasonably assess the time spent on various algorithms, we resize all images to 256x256, which provides stable output for cnhancements in later experiments.
5

---

Under review conference paper at ICLR 2020
(n)
(b)
(c)
(d)
(e)
(
Iaper
MSRCR
EDCP
UGAN
Ours
Figure 6: Qualitative comparisons for samples from real-world underwater image dataset RealA and RealB. (a)-() represents the samples selected from RealA and RealB.
Firstly, we compare the capabilities of different methods to improve the image visibility on the RealA, RealB, and RealC datasets. The qualitative comparison is shown Figure 5 and G. Most methods can improve the quality of images of slight haze effect. UCM, HE, and Fusion can enhance the brightness and contrast of the image, but are less uniform for color restoration and seem be over-enhanced in some areas of the image. The results of MSRCR appear to have suitable hue but lack sufficient saturation and contrast. UDCP and IBLA do not recover well for green-toned images, they make the image darker but enhance the contrast of the image. UGAN, WaterGAN can enhance the contrast of the image but they don't recover color well and generate some artifacts, which destroy the structural information of the image. The proposed method can recover the color of degraded underwater images while keeping proper brightness and contrast.
Table and Table 2 quantitatively show the scores of sample images in Figure 5 and Figure / respectively. Our proposed method has achieved the highest scores in (a), (c) and (f). In addition, the average quantized scores evaluated on RealA, RealB, and RealC datasets are shown in Table 3. Our model achieves the best

---

Under review conference paper at ICLR 2020
scores in terms of color restoration.
Table I: Quantitative UIQM values of samples in Figure The greater the UIQM values, the better the enhanced results, with blue representing the maximum and green representing the minimum.
Assessments Methods
(a)
(b)
(c)
(d)
(e)
(1)
(g)
(h)
(i)
Input
5,475
4.995
4.171
3.523
4.554
4.842
4.868
4.459
4.195
4.619
UCM
5.253
5.320
5.200
4.447
4.870
5.219
5.181
5.130
4.672
5.158
HE
5,080
5.369
4.814
4.779
4.907
4.925
5.174
5.215
4.493
5.247
MSRCR
4.047
4.636
5.229
4.135
4.516
4.528
4.465
4.259
4.022
4.684
Fusion
5.329
5.460
5.095
4.546
4.970
5.181
5.295
5.220
4.544
5.145
UIQM
UDCP
4.820
4.704
4.727
4.836
5.255
4.440
4.435
3.830
3.757
5,385
IBLA
5,468
5.302
3.867
3,559
20.606
4.861
4.941
3.537
3.659
4.999
UGAN
5.326
5.287
5,325
4.204
4.846
5.022
5.126
4.947
4.353
5.122
WaterGAN
5.024
4.934
4.833
2.763
4.594
4.414
4.547
4.879
3.953
4.700
Ours 5.602
5.387
5.379
4.219
4.820
5.327
4.868
5.110
3.922
5.018
The greater the UIQM values, the better the enhanced results, with blue representing the maximum and green representing the minimum.
Table 2: Quantitative UIQM values of samples in Figure 3
Assessments Methods
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
Input
4.865
4.316
4.923
4.516
3.854
4.837
3.740
4.320
4.819
3,468
UCM
5.127
5.093
4.999
5.165
4.558
5,056
4.028
4.925
4.773
4.634
HE
4.942
5.189
5.320
5,016
4.809
5.153
4.608
4.910
4.892
4.618
MSRCR
4.747
4.700
4.096
4.908
4.426
4.039
3.906
4.067
3.606
4.318
Fusion
5,280
5.131
5,063
5.184
4.485
5.030
4.023
4.811
4.961
4.557
UIQM
UDCP
5.389
5.256
4.932
4.868
4.731
5,406
4.722
5.180
4.962
5.131
IBLA
5.158
4.796
4.560
4.626
3.978
4.858
3,873
4.494
3.965
4.139
UGAN
5.249
5.185
5.040
4.800
4.832
5.026
4.561
5.601
4.934
4.675
WaterGAN
5.003
4.537
4.756
4.636
4.212
4.524
3.801
4.323
4.846
4.223
Ours 5.391
5.058
4.979
4.891
4.834
5.015
4.034
4.936
5.140
4.377
Table 3: Average quantitative UICM, UISM, UIConM and UIQM values real-world underwater image datasets RealA, RealB and RealC. The greater the values, the better the enhanced results, with blue representing the maximum
Datasets Assessments Input
UCM
HE
MSRCR Fusion
UDCP
IBLA
UGAN
WaterGAN Ours
UICM
-0.332
-0.059
0.003
-0.006
-0.127
-0.300
-0.233
-0.074
-0.079
0.006
UISM
7.151
7.092
7.194
6.934
7.000
7.073
7.148
7.045
6.820
7.096
RealA
UIConM
0.593
0.694
0.812
0.537
0.716
0.739
0.679
0.770
0.634
0.675
UIQM
4.22
4.574
5.027
3.967
4.622
4.721
4.533
4.832
4.280
4.508
UICM
-0.273
0.029
0.016
-0.006
-0.0350
-0.051
-0.193
-0.120
-0.151
0.091
UISM
7.169
7.053
7.120
6.944
6.910
7.080
7.049
6.957
6.821
6.992
RealB
UIConM
0.506
0.730
0.772
0.654
0.737
0.837
0.703
0.804
0.643
0.708
UIQM
3.920
4.695
4.864
4.387
4.675
5.080
4.590
4.927
4.309
4.598
UICM
-0,223
-0.023
-0.010
0.006
-0.110
-0.085
-0.136
-0.089
-0.121
0.044
UISM
7.310
7.309
7.312
7.348
7.318
7.428
7.305
7.117
6,895
7.282
RealC
UTConM
0.674
0.740
0.743
0.493
0.764
0.964
1.207
0.810
0.824
0.780
UIQM
4.561
4.803
4.816
3.932
4.891
5,636
6.469
4.996
4.979
4.942
UIQM is a non-reference assessment metric whose quantitative results depend largely on the value of scale factors. Structural information of images is not considered in these kinds of non-reference evaluation metrics. Although some enhanced images can get higher score, the visual quality poor, the reason is that the metric is calculated from the pixels. Therefore, we also employ three full-reference assessment metrics MSE, PSNR, and SSIM to evaluate the performance of different methods on synthetic datasets without training. The comparison results in Table 4 demonstrate that our proposed method achieves the best results in terms of MSE, PSNR, and SSIM.
7

---

Under review conference paper at ICLR 2020
The average inference time of different algorithms are compared in one computer with following configuration: Intel 17-875011 CPU, 16GB RAM, and GTX1060 6G GPU. The results are shown in Table Our model has the fastest processing speed compared to other methods. Moreover, the model we proposed methods. UGAN employs many
has the fewest Params and FLOPs compared other deep-learning-boet convolution layers with 512 kernels, which causes that there are too many network parameters. WaterGAN employs multiple networks, resulting in slow processing speed.
Table 4: Quantitative results evaluation on synthetic dataset by full-reference metrics: MSE, PSNR, SSIM
values. The smaller the MSE values, the greater the PSNR and SSIM values, the better the enhanced results,
with blue representing the best results
Datasets Assessments
Input
UCM
HE
MSRCR Fusion
UDCP
IBLA
UGAN
WaterGAN Ours
MSE
0.042
0.029
0.045
0.059
0.027
0.072
0.058
0.026
0.014
0.002
Synthesis
PSNR
20.68
23.46
18.315
13.25
23.13
17.37
19.10
20.63
20.25
30.31
SSIM
0.869
0.944
0.845
0.580
0.933
0.847
0.832
0.779
0.842
0.966
Table 5: Testing time and parameters of generator of different enhancement methods
UCM
HE
MSRCR
Fusion
UDCP
IBLA
UGAN
WaterGAN
Ours
Testing time (s)
1.284
0.009
0.076
0.118
2.051
4.561
0.022
10.347
0.008
Params (M)
54.41
28.62
1.93
FLOPs (M)
610
8053
3.8
As indicated by some previous works ([pplavikar P. 2019: Anwar S. 2019; Ding X, 2019), the performance
of high-level computer vision tasks (such as underwater target detection) on enhanced images is an indicator of image enhancement methods. We applied YOLO v3 (Redmon J. 2018) target detector underwater images and their enhanced versions generated by our model. The performance of underwater target detection is better enhanced versions degraded images. Figure 7 shows the results of YOLO v3 detector before and after processing the images with our model.
degraded
A
B
Figure 7: Underwater target detection results before and after enhancement. (A) Real-world underwater images and (B) output of our model for the real-world image. Red boxes represent scallops, blue boxes represent cucumbers, and green boxes represent urchins.
CONCLUSION
Based on an improved underwater imaging model, a generative adversarial network (UWGAN) for generating realistic underwater images is proposed in this paper. Then, U-net with combined loss functions is used for degraded underwater images enhancement. Our model is validated on both low-level and high- level underwater computer vision tasks, which demonstrate its effectiveness and robustness.
ACKNOWLEDGEMENT
The authors would like to acknowledge the National Key R&D Program of China (Grant No. 201 8YFC0309402) for funding this work.
8

---

Under review conference paper at ICLR 2020
REFERENCES
Liu Y Chan W H, Chen Q. Automatic white balance for digital still camera[J]. IEEE Transactions on Consumer Electronics, 1995, 41(3): 460-466.
Rizzi A, Gatta C, Marini D. Color correction between gray world and white patch[CJ//Human Vision and Electronic Imaging VII. International Society for Optics and Photonics, 2002, 4662: 367-375.
Pizer S M, Amburn P, Austin D, et al. Adaptive histogram equalization and its variations[J]. Computer vision, graphics, and image processing, 1987, 39(3): 355-368.
Ancuti C, Ancuti C O, Haber T, et al. Enhancing underwater images and videos by fusion[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012: 81-88.
He K, Sun J, Tang X. Single image haze removal using dark channel prior[J]. JEEE transactions on pattern analysis and machine intelligence, 2010, 33(12): 2341-2353.
Chiang JY, Y, Chen Y C. Underwater image enhancement by wavelength compensation and dchazing(J]. IEEE Transactions on Image Processing, 2011, 21(4): 1666
Krizhevsky A. Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural
Networks[[]iAAdvanee in neural information processing systems. 2012: 1097-1105.
Redmon J. Divvala S, Girshick R, et al. You only look once: Unified, real-time object
Octection[C|/PPooceedige of the IEEE conference computer vision and pattern recognition. 2016: 779. 788.
Kuehne H, Jhuang H, Garrote E, et al. HMDB: large video database for human motion recognitionn[]//201
International Conference on Computer Vision. IEEE, 2011: 2556-2563.
Anwar Li C, Porikli F. Deep underwater image enhancement[J], arXiv preprint arXiv: 807.03528, 2018.
Ancuti C, Ancuti O, De Vleeschouwer C. D-hazy: dataset to evaluate quantitatively dehazing [Lgor1th IEEE International Conference Image Processing (ICIP). IEEE, 2016: 2226-2230.
Uplavikar P, Wu Z, Wang Z. All-In-One Underwater Image Enhancement using Domain-Adversarial Learning[J]. arXiv preprint arXiv: 1905. 13342, 2019.
Goodfellow I, Pouget-Abadic J, Mirza M, et al. Generative adversarial nets[C]ViAdvances in neural information processing systems. 2014: 2672-2680.
Zhu J Y, Park Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C//Pooceeding of the IEEE international conference computer vision. 2017: 2223-2232.
Li Skinner K A, Eustice R M, et al. WaterGAN: Unsupervised generative network to enable real-time color correction of monocular underwater images[J]. IEEE Robotics and Automation letters, 2017, 3(1): 387-394.
Ronneberger O, Fischer P, Brox U-net: Convolutional networks for biomedical image CaaneentationCC//lttternall Springer, Cham, 2015: 234-241.
Conference Medical image computing and computer-assisted intervention.
Silberman N, Fergus R. Indoor scene segmentation using a structured light Sensor[CV//201 IEEE international conference on computer vision workshops (ICCV workshops). IEEE, 2011: 601-608.
Silberman N, Hoiem D, Kohli P, al. Indoor segmentation and support inference from rgbd [mages(CVV/Eurppen Conference Computer Vision. Springer, Berlin, Heidelberg, 2012: 746-760.
Li C, Guo C, Ren W, et al. An underwater image enhancement benchmark dataset and beyond[J]. arXiv preprint Xiv: 90119990000 00549, 2019.

---

Under review conference paper at ICLR 2020
Jerlov G. Marine optics[M]. Elsevier, 1976.
Iqbal K, Odetayo M. James A, et al. Enhancing the low quality images using unsupervised colour correction method[C]//2011 IEEE International Conference on Systems, Man and Cybernetics. IEEE, 2010: 1703-1709,
Hummel R. Image enhancement by histogram transformation[]]. Computer Graphics and Image Processing, 1977, 6(2):184-195.
Rahman Z, Jobson J. Woodell G A. Multi-scale 10etinex for color image ethancemenn(C)//nccceling of 3rd IEEE International Conference on Image Processing. IEEE, 1996, 3: 1003-1006.
Drews P. Nascimento E, Moraes F, et al. Transmission estimation underwater single Mmages[(//Proocedding the IEEE international conference on computer vision workshops. 2013: 825-830.
Peng Y Cosman C. Underwater image restoration based on image blurriness and light absorption[J].
IEEE transactions on image processing, 2017, 26(4): 1579-1594.
Fabbri C, Islam J. Sattar J. Enhancing underwater imagery using generative adversarial [ctworks[C]//0018 IEEE International Conference on Robotics and Automation (ICRA). IEFE, 2018: 7159-7165.
Panetta K, Gao C, Agaian Hluman-viiual-yyyte---nsippi underwater image quality measures[J]. IEEE
Journal of Oceanic Engineering, 2015, 41(3): 541-551.
Hore A, Ziou D. Image quality metries: PSNR vs. SSIM[C]//2010 20th International Conference on Pattern Recognition. IEEE, 2010: 2366-2369.
Anwar S, Li C. Diving Deeper into Underwater Image Enhancement: A Survey[J]. arXiv preprint arXiv:1907.0777 00766,,, 2019.
Ding X, Wang Y, Yan Y, et al. Jointly Adversarial Network to Wavelength Compensation and Dehazing of Underwater Images[J]. arXiv preprint arXiv:1907.055: 9055999, 2019.
Redmon J, Farhadi A. Yolov3: An incremental improvement[]], arXiv preprint wxXiv:1804002766 2018.
10

---

Under review conference paper at ICLR 2020
A LOSS FUNCTIONS
The most common loss function for image restoration is L2 error. However, which loss function is suitable for underwater image enhancement has been studied. Inspired by related article, the effect of different loss functions in U-net is studied in this paper. Table 0 shows the loss functions we used.
In mathematical formula, is an index of pixels in region X, (x) is pixel value in region X of image reconstructed by U-net and r(x) is the pixel value of corresponding ground truth. x the central pixel value of region X. Vx),), (x) respectively represent the gradient of reconstructed images and clear images. After several experiments and observations of the best reconstruction results, we set a to 0.8 this paper.
Table 6: Different loss functions for underwater image restoration. Including some basic loss functions and their combinations.
Name
Mathematical formula
Σ N
The L1 loss error
("
lx(x) r(x)
X
The L2 loss error
[²(X) = N
(g(x) - r
x
N Σ 1 - SX)
[SSIM
The SSIM loss error
xEX
The MSSSIM loss error
SMSM (X)
MS SSIM
N Σ (x)()) - vr(x)]
The GDL error
L (X) =
L1 + L2
[ILI2(X) = L + (1 a) L
L1 + SSIM
L (X) a SSIM + (1 a) L
L1 + MSSSIM
L SLLMS-SSIN (X) a MS- SSIM +
L
L1 + GDL
[u (X) a Loa (1 a) L
B
ABLATION STUDY
ieput
1.12
LI-SSMM
Figure 8: The visual quality of the sample image in RealC dataset with different loss functions. From (a) to (i) are respectively enhanced results of the loss function L1, L2, SSIM, MSSSIM, GDL, L1 + L2, L1 + SSIM, L1 + MSSIM, and L1 GDL.
Ablation study is mainly reveal the effects of different loss functions. We use different loss functions to train the network and test it on RealC, and synthetic datasets. The sample image is selected from the RealC dataset, as shown in Figure the enhanced results range from (a)~(i) are obtained with different loss functions, and images in the second row show the details in the red box area of the image. It can be seen from
11

---

Under review conference paper at ICLR 2020
the results in the second row, (a), (b), (f) appear striped artifacts. (c), (d), (g) cause color unevenness. The details of (e) are natural but it lacks sufficient saturation. (h), (i) show proper enhanced results, the color in (h) is more vivid but with slightly striped artifacts.
The enhanced result using the L1 or L2 loss function appears stripe-like artifacts while the SSIM or MSSSIM loss function causes color unevenness. The enhanced result of the GDL loss function is natural but lacks sufficient saturation. We calculated the MSE, PSNR, and SSIM metrics on the synthetic dataset. The quantitative scores Table 7 demonstrate that a combination of multiple loss functions can achieve better enhancement results.
Table 7: Quantitative results of different loss functions evaluation synthetic dataset by full-reference metries: MSE, PSNR, SSIM values.
Datasets
Assessments
Input
L.1
1.2
SSIM
MSSSIM
GDL
LI+L2
LI+SSIM
LI+MSSSIM
LI+GDL
MSE
0.0417
0.002
0.002
0.002
0.002
0.009
0.001
0.001
0.002
0.002
Synthesis
PSNR
20.68
29.91
28.72
29.99
28.27
25.21
32.97
32.82
30.31
30.81
SSIM
0.867
0.962
0,959
0.974
0.960
0.944
0.971
0.979
0.966
0.968
12


Reviews:
[{"review_id": "SyeZlLA6Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses U-net for underwater image restoration and enhancement. But, it is difficult to obtain realistic underwater images, thus this paper introduces a GAN-based method to generate realistic underwater images from in-air image and depth map pairs.\n\n- Although this paper points out that the previous work (i.e. WaterGAN) generates color noise and the camera model is not suitable, how does this proposed method overcome these points? Please make it clear.\n\n- The figures in this paper are too blurry to see them. To evaluate the effectiveness of the proposed method, the figures are important, thus, it would be better to make them clear.\n\n- The technical contribution of the proposed method is not clear. The proposed method seems to be just using the existing techniques.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Skl-8kPxjr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this article, the authors propose a generative adversarial network named UWGAN to generate realistic underwater images from the pairs of in-air images and depth images. Then, a U-Net was leveraged to enhance the results. \nHowever, the text suffers from too many language problems. The authors should consult professional proofreading services. As a courtesy towards referees, the quality of writing needs meticulous attention before a scientific paper should be submitted. \n\tOther comments:\n1.\tThe literature is limited. I found some novel works being done in the field that must be addressed and listed in the background and experiments.\n2.\tThe underwater imaging model presented in this paper derives from the Jaffe-McGlamery model, which is a common sense in this field. The authors use a generator to produce underwater images that only implements the common model by a neural network. Moreover, the statement of section 2.2 is not clear. Please rewrite this section.\n3.\tThe authors used U-Net without any improvement to enhance the results generated from UWGAN, which is the integration of existing models. \n4.\tThe authors claimed that their model is better than others, while there is no evidence to indicates that. For example, 1) in (page 5, line 4 from bottom), “It can be seen that our proposed method has achieved a higher score.”, can we observe this from the Table 1 and 2? 2) “The method we proposed has the fastest processing speed compared to other methods. Moreover, the method proposed in this paper has the fewest parameters compared to other deep-learning-based methods.”, it is suggested that a study about the parameters and FLOPs of the involved methods should be given.\n5.\tPlease carefully check the references. For example, “Hummel R. Image enhancement by histogram transformation[J]. Unknown, 1975.” lacks the journal name.\n6.\tHigh-resolution figures should be given in the manuscript.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gemwT6YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Update after rebuttal period]\nIn response, the authors cannot clearly clarify the difference between this work with existing works integrating the physical model into the network. Thus I stay my original score.\n\n\n[Original reviews]\nThis paper proposed an unsupervised generative adversarial network for underwater generating realistic underwater images and haze removal, which can simultaneously deal with the color restoration and haze in the realistic underwater environment.\n\nFirstly, according to the widely used physical model in the image processing area, employed the UnderwaterGAN to trained parameters in advanced, and then use U-Net for color restoration and haze removal of underwater images. However, many existing works used the physical model to represent the imaging principles and using deep network to learn prior knowledge. Thus, I think the proposed idea is a little bit incremental.\n\nFor the experimental part, the experimental results fully demonstrate the effectiveness of the proposed method in comparison with state-of-the-art methods. Additionally, the ablation studies in the appendix also give us the intuition by using the different loss functions. Also, I suggest the authors demonstrate the proposed method on not only low-level, but also high-level vision tasks, e.g., underwater image target detection. \n\nFinally, the paper is well organized and sentence expression is also clear, but small errors that are correctable. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed to improve the quality of underwater images, specifically color distortion and haze effect, by an unsupervised generative adversarial network (GAN). An end-to-end autoencoder network is used to demonstrate its effectiveness in comparing to existing works, while maintaining scene content structural similarity. Three reviewers unanimously rated weak rejection. The major concerns include unclear difference with respect to the existing works, incremental contribution, low quality of figures, low quality of writing, etc. The authors respond to Reviewers’ concerns but did not change the rating. The ACs concur the concerns and the paper can not be accepted at its current state.
Decision: Reject