Title: DIVIDE-AND-CONQUER ADVERSARIAL LEARNING FOR HIGH-RESOLUTION IMAGE ENHANCEMENT Anonymous

Abstract: This paper introduces a divide-and-conquer inspired adversarial learning (DA-CAL) approach for photo enhancement. The key idea is to decompose the photo enhancement process into hierarchically multiple sub-problems, which can be better conquered from bottom to up. On the top level, we propose a perception-based division to learn additive and multiplicative components, required to translate a low-quality image into its high-quality counterpart. On the intermediate level, we use a frequency-based division with generative adversarial network (GAN) to weakly supervise the photo enhancement process. On the lower level, we design a dimension-based division that enables the GAN model to better approximates the distribution distance on multiple independent one-dimensional data to train the GAN model. While considering all three hierarchies, we develop a multiscale training approach to optimize the image enhancement process, suitable for highresolution images, in a weakly-supervised manner. Both quantitative and qualitative results clearly demonstrate that the proposed DACAL achieves the state-ofthe-art performance for high-resolution image enhancement.

1. INTRODUCTION:

Despite many mobile camera technological advances we have today, our captured images often still come with limited dynamic range, undesirable color rendition, and unsatisfactory texture sharpness. Among many possible causes, low-light environments and under/over-exposed regions usually introduce severe lack of texture details and low-dynamic range coverage, respectively. Another critical issue is the amplification (during the enhancement process) of noise in the dark and/or texture-less regions, where the enhancement may not even be necessary. Due to these issues, images acquired under different conditions, or different parts of a single image, may require separate enhancement operations. In this context, customarily used context/content agnostic enhancement methods often lead to a poor performance in the overall visual assessment. Therefore, comprehensive methods that improve the perceptual quality of images are in high demand.

In recent years, deep image enhancement methods have demonstrated their superiority for color enrichment, texture sharpening as well as contrast adjustment. A large amount of works like (Dong et al. (2014) ; Kim et al. (2016) ; Mao et al. (2016) ; Shi et al. (2016) ; Wang et al. (2018b) ; Timofte et al. (2018) ; Nah et al. (2017) ; Zhang et al. (2017b; 2016) ; Yu et al. (2018) ; Wu et al. (2018) ; Yang et al. (2018) ; Ruixing Wang & Jia (2019) ) have made great success in one of such sub-tasks. However, in the real-world setting it is non-trivial to decompose the low-quality factors and treat them respectively from a low-quality images. Another family of approaches (e.g., Gharbi et al. (2017) ; Ignatov et al. (2017) ; Liu & Jung (2018) ; Huang et al. (2018) ; Chen et al. (2018) ) learns the enhancement as some non-linear mapping between low and high-quality image pairs, in a supervised manner. These methods require the desired high-quality images to be well aligned with the lowquality ones. Such image pairs are collected either by manually enhancing the low-quality images (using professional artists) or by using a calibrated rig of low and high-end cameras, with a very small baseline. Unfortunately, the data collection for these setups is not convenient, mainly due to the expensive manual retouching process and the physical limitation for small baseline. Moreover, fully-supervised methods often lack flexibility towards new domain adaptation, thus requiring lowhigh paired acquisitions for every low-end camera. Step 1#: divide divide divide

Step 2#: conquer

Step 3#: combine

Step To address the limitations of fully-supervised methods, the weakly-supervised deep photo enhancement methods (e.g., Ignatov et al. (2018) ; Chen et al. ( 2018)) have emerged recently. These methods merely require a set of target good quality images, whose contents and viewpoints share some similarity with those of the low-quality image domain, without requiring image acquisitions for the same scenes. On the collected unpaired low-and high-quality images, such kind of methods typically apply generative adversarial networks (GANs), which alternately optimizes its two components (i.e., generator and discriminator) using a min-max objective. The discriminator is optimized to approximate the desired-quality image distribution, and the generator is guided by the discriminator to learn the enhancement map. While achieving some success, they still have drawbacks when enhancing mixed visual perceptions or treating high-resolution images. Without the implementation of a pixelwise loss function in a supervised setup, it appears exceptionally difficult to simultaneously enhance multiple perceptual components (e.g., color, texture and illumination). Besides, due to the computational complexity and unstable adversarial training, the existing image enhancement methods treat high-resolution images using either the downscaled version or patch-wise processing strategies, neither of which are optimal. It is needless to mention that the downscaling loses images details, whereas, patch-wise enhancement does not maintain the spatial consistency over the whole image.

In this paper, inspired by the concept of the divide-and-conquer (DAC) paradigm, we aim at decomposing the original problem of high-quality image distribution learning into sub-problems, which can be better addressed from bottom to up. For this purpose, we propose a DAC inspired adversarial learning approach1 to better deal with the high-resolution photo enhancement problem. As depicted in Figure 1 , our basic learning process innovatively introduces the following three parts to the context of adversarial learning for high-resolution image enhancement.

• Divide: We hierarchically decompose the problem into three sub-problems: perceptionbased (additive and multiplicative maps), frequency-based (low and high frequencies), and dimension-based (multiple one dimensions) data distribution learning problems (from top to bottom). The third sub-problem that learns for each dimension separately is the basics for the original problem. In terms of implementation, these divisions are respectively carried out by: (a) perception-wise branching of the generator; (b) frequency-wise division of the discriminator's inputs; and (c) dimension-wise division of the discriminator's output.

• Conquer: To conquer the base sub-problems, we exploit an adaptive sliced Wasserstein GAN (AdaSWGAN) model. The basic idea of the AdaSWGAN is to first factorize highdimensional distributions into their multiple one-dimensional marginal distributions, followed by the approximation of each one-dimensional distributions independently. 2016)). Such works apply the multiplicative mapping E(x) = x * s -1 x (where s x is the multiplicative component) to address the problem of multiplicative noise removal or illumination improvement successfully. Accordingly, we introduce the perception-wise division to the generator so that the benefits of both additive and multiplicative map learning approaches can be fully exploited. Formally, the final map of the generator is

EQUATION

where β plays the trade-off between such two perception-based components2 and the operator '*' means element-wise multiplication.

To approximate the additive component r x and the multiplicative component s x , we design a twostream U-Net (Ronneberger et al. (2015) ) for the generator (i.e., photo enhancer). As shown in Figure 2 , we follow the standard U-Net design to apply the skip connection across the encoder and decoder3 . As skip connections concatenate local feature maps, they are good for reliable local feature extraction. In addition, we investigate that global information (e.g., scene category, object class or overall illumination) is useful for individual pixels to guide their local adjustment. Accordingly, we exploit a global concatenation operation to embed high-level information into decoded feature maps. While the encoder can transform input images into low-dimensional features, it does not always achieve a global feature (with a vector form) when the input images are of different sizes.

To address this issue, we additionally apply an average pooling operation with a fully convolutional layer to extract the global feature vector. As a result, this design enables our model to enhance fullresolution images while it is trained on down-scaled images. On top of the decoder, we design two different branches to learn the additive and multiplicative components separately. During the early training, we optimize the additive and multiplicative mappings alternately. In the end of training, we aggregate the resulting approximations from these two branches, followed by a group of regular convolutional network layers.

The idea of our designed two-stream U-Net is related to Chen et al. (2018) ; Ruixing Wang & Jia (2019) . For the additive component learning, Chen et al. (2018) 

2.2. FREQUENCY-BASED DIVISION:

For the supervised photo enhancement problem that requires corresponding ground-truth highquality images for guidance, the common reconstruction loss can be used to train the proposed deep enhancer. In contrast, the weakly-supervised photo enhancement task merely has a set of good quality images for reference, and thus for this task we need to learn the map from the low-quality image domain to high-quality image domain. One of the most promising approaches is cyclic GAN models (Zhu et al. (2017) ). Hence, we choose to adopt the cyclic GAN objective to train our developed enhancer:

EQUATION

where E : X → Y and Ê : Y → X are used for forward and backward image translation, respectively. They share the same design using the proposed two-stream U-Net. Here, C and Ĉ are the corresponding discriminators (or critics) which are used to guide the training of E and Ê, respectively. The GAN loss L GAN (E, C) and L GAN ( Ê, Ĉ) are first suggested to use the adaptive Wasserstein GAN loss (Chen et al. (2018) ) as,

EQUATION

where ŷ are random samples following the distribution P ŷ , which is uniformly sampled along straight lines between pairs of points sampled from P y and P E , and ∇ ŷ C( ŷ) is the gradient w.r.t ŷ.

Standard cyclic GAN models suggest to use the cyclic consistency loss and identity mapping loss. We follow (Chen et al. (2018) ) to define these two losses based on mean square error (MSE):

EQUATION

To further decompose the perception-based distribution learning, we follow (Ignatov et al. (2017; 2018) ) to apply frequency-based division to the discriminator C. Specifically, we adopt the separation of enhancing on low frequencies (e.g., main structures, major contents and colors), and optimizing on high frequencies (e.g., textures and small details). For this purpose, we replicate the discriminators to specialize on various frequencies of input images. Without loss of generality, we study two frequencies for discriminators: For low frequency data, we utilize the RGB image blurred by a Gaussian kernels, whereas, the grayscale component of the discriminator input images is used as the high frequency data. Both are equal in architecture and parameter values, and their GAN losses are averaged in the end. In this case, individual discriminators are encouraged to focus on an easier distribution approximation task, and thus become more reliable to supervise the training of the proposed photo enhancer.

2.3. DIMENSION-BASED DIVISION AND SOLUTION:

While the frequency-based division for the discriminator's input can reduce the complexity of training Wasserstein GAN to some extent, it still suffers from the unstable training issue when the raw images lie in a high dimensional space, where we can only select samples very sparsely during training. To address this issue, we suggest to further decompose the challenging estimation of a highdimensional distribution into simpler estimation of multiple one-dimensional distributions, which is 149

EQUATION

)

When ∇ ŷ C( ŷ) is larger than a fixed upper bound, it reflects the current weight λ is too small and the the underlying idea of sliced Wasserstein distance computation. To achieve this goal, we exploit an adaptive version of sliced Wasserstein GAN (AdaSWGAN) loss:

min E max C θ∈S n-1 E y∼Py [C(y)] -E ỹ∼P E [C(E(x))] + λE ŷ∼P ŷ [max(0, ∇ ŷ C( ŷ) 2 -1)],

(5) where θ is the orthogonal projection matrix for the decomposition from high-dimensional data to independent one-dimensional ones under C, ŷ denotes random samples following the distribution P X which is sampled uniformly along straight lines between pairs of points sampled from P X and P G , and ∇ ŷ C( ŷ) is the gradient with respect to ŷ. As the parameter λ weights the gradient penalty, it can highly influence the trade-off between the original sliced Wasserstein distance and the Lipschitz constraint. To reach a good balance, we adopt an adaptive learning manner to update the hyperparameter by observing the moving average of gradients:

EQUATION

where λ is the weight for gradient penalty, and η is the decay constant. When ∇ ŷ C( ŷ) is larger than a fixed upper bound τ , the relative influence of λ becomes too small and therefore the penalty is not strong enough to satisfy the Lipschitz constraint. In this case, the weight λ should be increased with a certain scale (in our experiments, we always set 2 times bigger). Otherwise, λ is decreased to half of the current value.

During training, the parameterized projection matrices θ should remain orthogonal. For this requirement we first initialize the parameters with random orthogonal matrices through QR decomposition, then update them on a Stifel manifold during training. Particularly, we follow (Wu et al. (2019) ) to optimize the orthogonal matrices θ on Stiefel manifolds.

Few variations of sliced Wasserstein GANs (Deshpande et al. (2018) ; Ishan Deshpande (2019); Wu et al. (2019) ) are proposed in the recent years. Differently from (Deshpande et al. (2018) ; Ishan Deshpande ( 2019)), we use the parameterized projection matrix to optimize the sliced Wasserstein distance approximation. When compared to (Wu et al. (2019) ), we adopt an adaptive weight scheme to adjust the penalty weight. To show the advantage of the proposed AdaSWGAN, we follow the sampling strategy of (Gulrajani et al. (2017) ) to generate 25 Gaussian distributed data. On the toy dataset, we compare it against the four existing Wasserstein GAN models4 . The results in Figure 3 show that the proposed AdaSWGAN has a better convergence to fit the toy data than the other compared methods.

3. MULTISCALE NETWORK AND TRAINING:

To improve the enhancement on high-resolution images, we introduce a multiscale enhancer that is trained on images with various resolutions. This design allows for a wider range of learned receptive fields, from coarse to fine. The multiscale enhancer consists of various enhancers at multiple scales.

Let's take a two-scale case for example. The two enhancers are denoted as E l , E h , which enhance images at resolution of 512×256 and 1024×512, respectively. The multiscaled design of enhancers is expected to effectively aggregate the visual features over various scales.

The high-scale enhancer E h has the same structure as the low-scale enhancer E l , which is shown in Figure 2 . To transmit the low-scale information smoothly from the low-scale to the high-scale components, the high-scale enhancer also inherits the learned parameters from the low-scale enhancer. In addition, it repeats some more layers on the bottom and the top of the low-scale enhancer. Specially, it adds one more down-sampling component including convolution, SELU, and batch normalization at the bottom. One more up-sampling component is applied to the top. It contains an up-scale resizing and a convolution operation. Following the lower-scale enhancer design, we apply the same two-stream design to learn the additive and multiplicative components for the high-resolution image enhancement as well.

It is known that synthesizing high-resolution images yields significant challenges to the GAN training. For example, differentiating high-resolution real and generated images requires the discriminator to have a deeper architecture or convolutions with a larger receptive field. But such two solutions will often lead to overfitting and out of memory issues. Besides, due to the memory limit, we have to use smaller minibatches that will further harm training stability. Therefore, we also follow the design of multiscale enhancer to exploit multiscale discriminators for more reliable training guidance on different resolutions. Similarly, we insert some more layers into the bottom of the lower-scale discriminator to construct higher-scale discriminators for more reliable training guidance on higherresolutions.

For more details on the design of the multiscale enhancer and discriminator, please refer to the appendix. We train the proposed multiscale enhancer with three stages. At the first stage, we train the lower-enhancer E l using either reconstruction loss in the supervised case or using the proposed frequency-and dimension-based GAN loss for the weakly-supervised case. At the second stage, we tune the parameters of the newly added layers in the higher-scale enhancer E h while fixing the parameters of the lower-scale enhancer E l . Finally, we train the multiscale enhancer as a whole.

To 2018)). In contrast, our multiscale enhancer is built upon a two-stream U-Net based model, which is tailored for better high-resolution image enhancement.

4. EXPERIMENT:



4.1. SUPERVISED PHOTO ENHANCEMENT:

This task aims at learning the enhancement mapping between low-and high-quality image pairs in a supervised fashion. We use the commonly-used MIT-Adobe FiveK (Bychkovsky et al. (2011)) dataset for this task. The dataset consists of 5,000 high-resolution (higher than 1K) images, each of which is retouched by five experts. For the evaluation, we use 2,250 images and their retouched versions from photographer C for training, and the remaining 498 images are used for testing. Distinct from most of existing works, we go for high-resolution image enhancement. To study this task, we compare our proposed adersarial learning approach of divide-and-conquer (DACAL) against stateof-the-art methods including White-Box (WB) (Hu et al. (2018) ), Distort-and-Recover (DR) (Park et al. (2018) ), DSLR-Photo Enhancement (DPED) (Ignatov et al. (2017) ), and the supervised version of the Deep Photo Enhancement (DPE) method (Chen et al. (2018) ). Except DPED, the rest three methods are originally designed for down-scaled image enhancement. In contrast, our model can be trained on multiscaled images. For details on our training details5 , please refer to the appendix. As the dataset has paired original/retouched images, we use both the Peak Signal-to-Noise Ratio (PSNR) and the multi-scale structural similarity index measure (SSIM)6 (Wang et al. (2003) ) to evaluate the comparing methods.

Table 1 summarizes the PSNR and SSIM results on both downscaled images of size 512 × 256 and full-resolution ones. Compared to the state-of-the-art methods, our proposed DACAL achieves the highest PSNR and SSIM. Note that our DACAL achieves the highest PSNR7 (24.15 dB) on the MIT-Adobe FiveK dataset. In particular, we discover the PSNR and SSIM values of some competing methods like DPE decline by a certain margin when enhancing full-resolution images.

In contrast, our proposed DACAL has relatively stable PSNR and SSIM values in the high-res case. For a quantitative perceptual quality comparison, we employ 20 Amazon Mechanical Turk (MTurk) workers to compare 100 image pairs, which are from our DACAL and the competing methods respectively. As shown in Table 3 , the highest preference score shows the superiority of our DACAL.

In addition, we conduct the ablation study on the proposed two-stream U-Net. Specially, we evaluate the separative use of additive and multiplicative streams denoted by DACAL l1 and DACAL l2 , respectively. As shown in Table 1 , they are outperformed by our two-stream version DACAL l . We also evaluate the performance by cascading these two streams: additive cascaded by multiplicative stream, or multiplicative cascaded by stream. As we find the former case works better than the later one in terms of PSNR and SSIM. Here, we only report the results of the former one DACAL l3 . The improvement of our suggested two-stream version further justifies its effectiveness.

In Figure 4 , we also study the visual results of the two components and their resulting maps. The results reflect that the additive one targets for better color and texture, and the multiplicative one targets on enhancing illumination. The result of final map shows the benefit of fusing additive and multiplicative streams.

4.2. WEAKLY-SUPERVISED PHOTO ENHANCEMENT:

This task performs photo enhancement using one set of good-quality images instead of paired before/after images. For this task, we use the DPED dataset (Ignatov et al. (2017) ) and our collected Huawei-Flickr dataset. The DPED dataset is composed of images from smartphone cameras (i.e., iPhone 3GS, BlackBerry Passport and Sony Xperia Z) paired with images of the same scenes captured by a DSLR camera (i.e., Canon 70D). We use the iPhone-DSLR data (5,614 iPhone images of size 2048 × 1536, 5,902 DSLR images of size 3648 × 2432) to evaluate the competing methods on the full-resolution images in a weakly-supervised manner. As the dataset also contains 4,353 wellaligned 100 × 100 iPhone-DSLR image patch pairs, we use its testing image patches to compute the PSNR and SSIM values for quantitative evaluation. As for our Huawei-Flickr dataset, we collect 5,138 images of size 1920 × 1080 by Huawei Mate20 pro and use the downloaded 604 HDR Flickr images with around 1K resolution from (Chen et al. (2018) ) for the target.

Table 2 reports the PSNR and SSIM results of the evaluated methods on the DPED dataset. When compared with the state-of-art weakly-supervised methods WESPE (Ignatov et al. (2018) ) and DPE (Chen et al. (2018) ), our DACAL reaches the highest PSNR (20.90 dB) and SSIM. The table also shows the improvement of the introduced frequency based GAN and sliced SWGAN loss. For a quantitative perceptual quality comparison on our DACAL, WESPE and DPE, we ask 20 MTurk works for user study on 120 image pairs for DPED and Huawei-Flickr. The results in Table 3 show that our method achieves much higher preference score than all competing methods in terms of the sampled human perception. The qualitative results from Figure 5 reflect that our DACAL has the best performance in terms of color rendering, texture sharpening, and illumination adjustment for the DPED iPhone images. In Figure 6 , we present enhancement results for our Huawei-Flickr dataset. We can find that WESPE tends to make the image over-exposed, while DPE has serious texture issues due to its design for low-resolution image enhancement. By comparison, our DACAL performs most promisingly for color, texture, and light condition improvements.

5. CONCLUSION:

This paper addresses mix-perception enhancement of images with a divide-and-conquer inspired adversarial learning approach, which divides and merges perception-based, frequency-based and dimension-based sub-problems. For high-resolution image enhancement, we suggest to progressively grow the proposed multiscaled model along various resolutions. Evaluations on both su- 

:

There exist some networks like(Romaniuk & Hall (1993);Ghosh et al. (2017);Nowak et al. (2018)) applying the paradigm of DAC to general neural networks. While we found two concurrent works(Kim et al. (2019); Lin et al. (2019)) introduce the DAC concept to adversarial networks, they merely study the decomposition of generator. In contrast, our approach applies the DAC concept to the whole adversarial learning process.

Our experiments were conducted with β = 1, which can be chosen differently for known sub-tasks priors.

The encoder includes groups of convolution, SELU(Klambauer et al. (2017)) and batch normalization(Ioffe & Szegedy (2015)), while the decoder consists of convolution, resizing operation based layers.

For fair comparison, same generator and discriminator architectures (ofGulrajani et al. (2017)) were used for all compared methods

The code will be released once the paper is published.

For short, we denote the mutli-scale SSIM as SSIM throughout the paper.

Recently the CVPR19 paper (RuixingWang & Jia (2019)) has corrected their model's PSNR from 30.80 dB to 23.04 dB, on its GitHub webpage.

Figures and Tables:

Figure 1: Conceptual illustration of the proposed adversarial learning approach of divide-andconquer (DACAL). It decomposes the high-quality image distribution learning problem into perception-, frequency-and dimension-based sub-problems in a hierarchical structure. Following the division, the sub-problems are conquered and combined in a deep adversarial training fashion.

Combine: Our model combines the solutions from bottom to top by, (a) aggregating the approximated one-dimensional distributions to obtain the frequency-based solutions; (b) averaging frequency-based solutions to obtain the perception-based solutions; (c) summing two perception-based solutions to learn the high-quality image distribution.

Figure2: Illustration of the proposed network design for our adversarial learning approach of divideand-conquer. Specially, we apply the perception-based division to the generator, the frequencybased one to the discriminator input, and the dimension-based one to the discriminator output.

Figure 3: Results of WGAN-GP, AdaWGAN-GP, SWGAN-GP and the proposed AdaSWGAN-GP (three rows indicate training for 1000, 2500 and 5000 iterations resp.) for the 25 Gaussians dataset it would highly influence the trade-off between the original sliced Wasserstein distance and the 147 Lipschitz constraint. To achieve a good balance, we adopt an adaptive learning manner to update the 148 hyperparameter by observing the moving average of gradients:

Figure 3: Results of WGAN (Gulrajani et al. (2017)), AdaWGAN (Chen et al. (2018)), SWGAN (Wu et al. (2019)) and the proposed AdaSWGAN (three rows indicate training for 1000, 2500 and 5000 iterations resp.) for the 25 Gaussians dataset. The orange points represent the target distribution, green points are the generated samples, and the curves indicate the value surfaces of the discriminators of compared models.

overcome the unstable GAN training issue on high-resolution images, there exist various cascaded or multiscaled network designs like (Denton et al. (2015); Dosovitskiy & Brox (2016); Gatys et al. (2016); Johnson et al. (2016); Zhang et al. (2017a); Huang et al. (2017); Karras et al. (2017); Wang et al. (2018a)) that synthesize high-resolution images. However, such works rarely study the multiscaled network based on the U-Net design, which has shown its advantages over other networks for image enhancement(Chen et al. (2018);Huang et al. (

Figure 4: Visual results of additive, multiplicative components and their combination. It is best to zoom in on the high-resolution pictures.

Figure 5: High-resolution image enhancement results for the DPED iPhone-DSLR data. It is best to zoom in on the high-resolution pictures.

Figure 6: High-resolution image enhancement results for our Huawei-Flickr data. It is best to zoom in on the high-resolution pictures.

Figure 7: Illustration of the used multi-scaled enhancer (a) and multi-scaled discriminator (b) for our proposed method

introduces a global U-Net. Compared to our model, the global U-Net fromChen et al. (2018) can only accept images of a fixed low-resolution due to the hard design (a fixed filter for the fully-connected layer) for the global feature extraction. For example, when feeding 8 × 8 × 128 feature map into the global feature extraction, it has to use 8 × 8 filters to obtain the global feature 1 × 1 × 128. For different image sizes, the resulting embedded feature size will be different, say 4 × 8 × 128. In such cases, the global feature extraction will fail. For the multiplicative component learning, RuixingWang & Jia (2019) designs a bilateral-filtering based network, while our enhancer adopts regular convolutions and resizing operations.

Table content: <table/>

PSNR and SSIM results for the MIT-Adobe FiveK(Bychkovsky et al. (2011)) test images. Here, WB and DR indicate the White-Box and Distort-and-Recover methods, respectively. DACAL l1 , DACAL l2 , DACAL l3 and DACAL l represent the use of individual additive, individual multiplicative, multiplicative cascaded by additive, and our suggested parallel fusion (two-stream strategy), respectively. DACAL h is our higher-scale version. PSNR d /SSIM d and PSNR f /SSIM f indicate the results on downscaled images and full-resolution images, respectively. DACAL l 2 DACAL l 3 DACAL l DACAL h

Table content: <table><tr><td colspan="4">WB DACAL l 1 PSNR d 18.86 21.64 DR DPED DPE 21.05 22.10 22.73 PSNR f 19.09 21.52 20.86 21.65 22.43</td><td>22.99 22.69</td><td>23.01 23.02</td><td>23.52 23.56</td><td>24.15 24.07</td></tr><tr><td>SSIM d 0.928 0.936 SSIM f 0.920 0.922</td><td>0.922 0.916</td><td>0.947 0.894</td><td>0.958 0.948</td><td>0.942 0.942</td><td>0.949 0.940</td><td>0.959 0.954</td><td>0.962 0.956</td></tr></table>

PSNR and SSIM results for the DPED(Ignatov et al. (2017)) test 100 × 100 image patches. Here, l, f, d for DACAL represent the use of our proposed perception-based, frequency-based and dimension-based division respectively. DACAL h is our higher-scale version.

Table content: <table><tr><td>PSNR100 SSIM100</td><td>WESPE 17.45 0.854</td><td>DPE 18.53 0.861</td><td>DACAL l DACAL l+f DACAL l+f +d DACAL h 19.62 20.01 20.43 20.90 0.868 0.869 0.872 0.874</td></tr></table>

Preference ratio (one vote is lost for DACAL vs. Input) from MTurk user study

Table content: <table><tr><td>DACAL vs. Adobe DPED Huawei-Flickr 366:34 Input 395:4 384:16</td><td>White-Box Distort-and-Recover DPED WESPE 389:11 303:97 365:35 N/A N/A N/A N/A 384:16 N/A N/A N/A 341:59</td><td>DPE 370:30 344:56 360:40</td></tr></table>

Peer review (iclr2020):
The authors propose a divide-and-conquer style adversarial learning algorithm for weakly supervised learning of high resolution image enhancement. Their work is primarily motivated by the limitations of modern mobile cameras in producing images with adequate dynamic range, texture sharpness, and exposure. They also point out limitations of recent work in image enhancement; e.g. supervised approaches suffer from difficulties in data collection (must have paired low quality, high quality images) and previous applications of GANs have suffered from their inability to simultaneously enhance multiple *perceptual* qualities of the input images (i.e. texture, color, light, etc). Furthermore, many recent approaches rely on downsampling or patching strategies in order to deal with the computational demands of high resolution images. The authors aim to address these limitations by 1) introducing a *divide and conquer* approach in the generator/discriminator networks, thereby jointly optimizing a range of image qualities, 2) adopting the CycleGAN objective for weakly-supervised learning of low/high image sets, and 3) proposing a multi-scale architecture that is able to transfer learned features on downsampled low-resolution images to higher resolution images and improve the overall quality of results. They perform an empirical study using several datasets comparing PSNR and SSIM scores with previously proposed methods. They also provide Mechanical Turk human evaluation results for a small sample of images from two of the datasets. The fundamental idea of divide-and-conquer for image enhancement, as well as the multi-scale architecture, are both interesting contributions to the field. The authors also demonstrate fairly promising results on several datasets. However, due to the fact that this work ambitiously introduces a wide range of architectural changes, it suffers from a general lack of thoroughness in empirical support for all of the proposed ideas. For example, while the authors provide reasonable and convincing support for the divide and conquer approach through an ablation study, they fail to provide adequate justification for other proposals, such as the *adaptive* sliced Wasserstein GAN (dubbed AdaSWGAN). The only empirical evidence given for this change is an unconvincing qualitative comparison on the 25-Gaussians dataset, in which the alleged improvement over previous methods is unclear. Additionally, despite the fact that the proposed multi-scale architecture is solving a problem suspiciously similar to super-resolution, the authors do not compare their method to similar multi-scale, super-resolution methods such as ESR-GAN. The authors also do not provide any runtime analysis (theoretical or empirical), which seems fairly critical given the apparent complexity of their method. Given these concerns, as well as some others which I will detail per-section below, I am giving this paper an overall score of *weak reject* for lack of clarity and rigor. If the authors are able to address some of my questions and concerns, I am willing to increase this score, as I do think that some of their proposed ideas are valuable contributions. Section 1: Introduction 1. *Besides, due to the computational complexity and unstable adversarial training, the existing image enhancement methods treat high-resolution images using either the downscaled version or patch-wise processing strategies, neither of which are optimal* Unless I have misunderstood something about the authors* multi-scale architecture, it seems that their method (as applied in the experiment) also suffers from this limitation. In the appendix, they say that the high-resolution enhancer uses images of size 1024x512 resolution, which appears to be smaller (and oddly has a different aspect ratio) than the resolution of their training data. They note that it is *still feasible* to train their enhancer on higher-resolution images *if [they] have more computational resources.* But the same would also apply to other methods, since the whole reason for downscaling/patching is to lessen these computational constraints. 2. It is not clear how figure 1 and figure 2 relate to one another. The generator does not perform frequency-based or dimension-based division, so it seems that this *recombining* step is sort of implicit in the loss function learned by the discriminator. This should be clarified. Section 2: Proposed Method 1. Please clarify the x*s_x^-1 term in equation (1); specifically, what is the meaning of the -1 exponent? Is it simply 1/s_x? If so, why is the reciprocal necessary? 2. The mathematical meaning of global concatenation is not clear from the figure or the text. Please clarify. 3. There are no citations or theoretical justication for the section on conv + average pooling with regards to capturing meaningful global features. Specifically the claim *this design enables our model to enhance full resolution images while it is trained on downscaled images* needs empirical and/or theoretical support. This seems more like conjecture than fact. 4. Organization of section 2.2 is weird. It*s supposed to be about *frequency based division* but most of the section is dedicated to discussing CycleGANs. Perhaps these topics should be split into separate sections. 5. For freqency separation, is a Fourier transform used? Or just the application of Gaussian kernels and grayscale components as described in the last paragraph? Additionally, how are the Gaussian kernels parameterized? 6. The integral in equation (5) is missing a differential, presmuably d-theta. The theta in S^{n-1} notation doesn*t really make sense for integrals, since the space is (I think) continuous. If not continuous, this should be a summation. 7. As mentioned before, the motivation for an adaptive lambda parameter in the Wasserstein objective is not clear. Does this architecture not converge or perform worse without it? 8. The adaptive lambda also trades one hyperparameter for another, i.e. the bound tau. How is this term chosen? And what is the effect on performance? 9. The 25-Gaussians experiment is does not provide adequate support for the necessity of AdaSWGAN. The source paper for SWGAN provides a far more comprehensive comparison as well as quantitative results, rather than a single visualization. Furthermore, there is no evidence given that AdaSWGAN has better performance/convergence on real data. 10. Typo in second to last paragraph: Stifel -> Stiefel Section 3: Multiscale Network and Training 1. Please clarify whether or not the given low/high resolution image sizes in the appendix are used by the DACAL architecture in all experiments. 2. This section should be condensed to the main points. Some of the details and tangential discussions can be moved to the appendix. Section 4: Experiment 1. Please clarify the low/high image resolutions for the MIT-Adobe FiveK datasets and whether or not the images used for the high resolution DACAL are downscaled. 2. How are SSIM_f and PSNR_f calculatd on full resolution images if DACAL (and the other methods) operate on downscaled version of the images? Is there an upsampling operation applied in post processing? 3. Why is DPED in the weakly supervised section if it contains paired images? Is that not a supervised task? 4. Space permitting, please provide a qualitative example from one of the supervised tasks that shows the enhanced output of DACAL vs. the ground truth high resolution image. 5. Is the CycleGAN architecture still used for supervised learning tasks? Or just the two-stream U-Net? 6. Style suggestion: replace *our DACAL* with just *DACAL* throughout this section. This is better technical writing practice. 7. As mentioned before, please provide some assessments of computation costs and runtime (theoretical or empirical) for DACAL vs. the other methods. If DACAL requires significantly more compute time or memory, this is an important trade-off to consider. Update 1: Fixed typos