SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Methods	#Params	MIT-FiveK (480p)	MIT-FiveK (original)
PSNR↑	SSIM↑	LPIPS↓	△E↓	PSNR↑	SSIM↑	LPIPS↓	△E↓
UPE [35]	999K	21.82	0.839	0.136	9.16	20.41	0.789	0.253	10.81
HDRNet [12]	482K	23.31	0.881	0.075	7.73	22.99	0.868	0.122	7.89
CSRNet [14]	37K	25.31	0.909	0.052	6.17	24.23	0.891	0.099	7.10
DeepLPF [26]	1.72M	24.97	0.897	0.061	6.22	N.A.	N.A.	N.A.	N.A.
LUT [38]	592K	25.10	0.902	0.059	6.10	23.27	0.876	0.111	7.39
sLUT [36]	4.52M	24.67	0.896	0.059	6.39	24.27	0.876	0.103	6.59
CLUT [39]	952K	24.94	0.898	0.058	6.71	23.99	0.874	0.106	7.07
Ours	731K	25.53	0.910	0.055	5.64	24.52	0.897	0.081	6.34
Table 2: Quantitative comparison on MIT-Adobe FiveK [4] dataset. "N.A." means that the results are not available due to insufficient memory of the GPU.
Datasets: We evaluate the performance of our network on two challenging benchmark datasets: MIT-Adobe FiveK [4] and  $\mathrm{HDR + }$  burst photography [13]. The MIT-Adobe FiveK dataset is widely recognized as a benchmark for evaluating photographic image adjustments. This dataset comprises 5000 raw images, each retouched by five professional photographers. In line with previous studies [38, 36, 39], we utilize the ExpertC images as the reference images and adopt the same data split, with 4500 image pairs allocated for training and 500 image pairs for testing purposes. The  $\mathrm{HDR + }$  dataset is a burst photography dataset collected by the Google camera group to research high dynamic range (HDR) and low-light imaging on mobile cameras. We post-process the aligned and merged frames (DNG images) into 16-bit TIF images as the input and adopt the manually tuned JPG images as the corresponding reference images. We conduct experiments on both the  $480\mathrm{p}$  resolution and 4K resolution. The aspect ratios of source images are mostly 4:3 or 3:4.
Evaluation metrics: We employ four commonly used metrics to quantitatively evaluate the enhancement performance on the datasets as mentioned above. The  $\triangle E$  metric is defined based on the  $L_{2}$  distance in the CIELAB color space. The PSNR and SSIM are calculated by corresponding functions in skimage.metrics library and RGB color space. Note that higher PSNR/SSIM and lower LPIPS/ $\triangle E$  indicate better performance.
Implementation Details: To optimize the network, we employ the Adam optimizer [18] for training. The initial values of the optimizer's parameters,  $\beta_{1}$  and  $\beta_{2}$ , are set to 0.9 and 0.999, respectively. The initial learning rate is set to  $2\times 10^{-4}$ , and we use a batch size of 1 during training. In order to augment the data, we perform horizontal and vertical flips. The training process consists of 200 epochs. The implementation is conducted on the Pytorch [31] framework with Nvidia Tesla V100 32GB GPUs.
3.2 Quantitative Comparison Results
In our evaluation, we comprehensively compare our proposed network with state-of-the-art learning-based methods for tone mapping in the camera-imaging pipeline. The methods included in the comparison are UPE [35], DeepLPF [26], HDRNet [12], CSRNet [14], 3DLUT [38], spatial-aware 3DLUT [36], and CLUT-Net [39]. To simplify the notation, we use the abbreviations LUT, sLUT, and CLUT to represent 3DLUT, spatial-aware 3DLUT, and CLUT-Net, respectively, in our comparisons. It is important to note that the input images considered in our evaluation are 16-bit uncompressed images in the CIE XYZ color space, while the reference images are 8-bit compressed images in the sRGB color space.
Among the considered methods, DeepLPF and CSRNet are pixel-level methods based on ResNet and U-Net backbone, while HDRNet and UPE belong to patch-level methods, and LUT, sLUT, and CLUT are the image-level methods. Our method also falls within the image-level category. These methods are trained using publicly available source codes with recommended configurations, except
UPE
HDRNet
CSRNet
sLUT
CLUT
Figure 4: Visual comparison with state-of-the-art methods on a test image from the  $\mathrm{HDR + }$  dataset [13]. The error maps in the upper left corner facilitates a more precise determination of performance differences. Best viewed in color and by zooming in.
DeepLPF
for sLUT. Since the training code and weights of this method have never been released, we reproduce the results according to the description in the released article.
Tab. 1 presents the quantitative comparison results on the  $\mathrm{HDR + }$  dataset for two different resolutions. Notably, our method exhibits a significant performance advantage over all competing methods on both resolutions, as indicated by the values highlighted in bold across all metrics. Specifically, our method achieves a notable  $0.49\mathrm{dB}$  improvement in PSNR compared to the second-best method, sLUT [36], at  $480\mathrm{p}$  resolution. This advantage becomes even more pronounced (1.25dB) when operating at the original image resolution, demonstrating the robustness of our approach for high-resolution images. Similarly, when evaluated on our second benchmark, the MIT-Adobe FiveK dataset (refer to Tab. 2), our method consistently demonstrates a clear advantage over all competing methods. However, for all methods, the FiveK dataset offers limited improvements compared to the  $\mathrm{HDR + }$  dataset, which can be attributed to two main reasons. Firstly, some reference images in the FiveK dataset suffer from overexposure or oversaturation, presenting challenges for the enhancement methods. Secondly, inconsistencies exist in the reference images adjusted by the same professional photographers, leading to variations between the training and test sets. More discussions can be found in the supplementary material.
3.3 Qualitative Comparison Results
To evaluate our proposed network intuitively, we visually compare enhanced images on the two benchmarks, as shown in Fig. 4 and Fig. 5. Note that the input images are 16-bit TIF images, which regular display devices can not directly visualize; thus, we compress the 16-bit images into 8-bit images for visualization. These figures show that our proposed network consistently delivers visually appealing results on the MIT-Adobe FiveK and  $\mathrm{HDR + }$  datasets. For example, in Fig. 4, our method excels in preserving intricate details such as tree branches and grass texture while enhancing brightness. Moreover, our results exhibit superior color fidelity and alignment with the reference image. Similarly, in Fig.5, while other methods suffer from poor saturation in the shaded area of the reflected building, our method accurately reproduces the correct colors, resulting in a visually pleasing outcome. These findings highlight the effectiveness and superiority of our method in tone mapping tasks. More visual results can be found in the supplementary material. Since the central goal of the tone mapping task is to primarily recalibrate the tone of the image while compressing the dynamic range, the visual differences between the results produced by the various state-of-the-art methods are minimal. To intuitively demonstrate the visual differences, we utilize the error maps to facilitate a more precise identification of performance differences.
CLUT
Ours
Figure 5: Visual comparison with state-of-the-art methods on a test image from the MIT-Adobe FiveK dataset [4]. The error maps in the upper left corner facilitates a more precise determination of performance differences. Best viewed in color and by zooming in.
Metrics	Framework Component	Low-frequency Image Resolution
Baseline	+ Weight Map	+ Transformer	+ Learnable Filter	64 × 64	128 × 128	256 × 256
PSNR↑	23.16	24.41	(+1.250)	25.34	(+0.930)	26.62	(+1.280)	26.62	26.69	(+0.070)	26.81
SSIM↑	0.842	0.856	(+0.014)	0.868	(+0.012)	0.907	(+0.039)	0.907	0.909	(+0.002)	0.913
LPIPS↓	0.113	0.111	(-0.002)	0.101	(-0.010)	0.063	(-0.038)	0.063	0.061	(-0.002)	0.058
△E↓	7.04	6.23	(-0.81)	5.92	(-0.31)	5.31	(-0.61)	5.31	5.29	(-0.02)	5.25
Table 3: Ablation study of framework component and the selection of pyramid layers. All four metrics are reported.
3.4 Ablation Study
Break-down Ablations. We conduct comprehensive breakdown ablations to evaluate the effects of our proposed framework. We train our framework from scratch using paired data from the  $\mathrm{HDR + }$  dataset [13] and evaluate its performance on the  $\mathrm{HDR + }$  test set. The quantitative results are presented in Tab. 3. We begin with the baseline method, 3D LUT [38], without utilizing pixel-level weight maps or learnable local Laplacian filters. The results show a significant degradation, indicating the insufficiency of 3D LUT. When pixel-level weight maps are introduced, the results improve by an average of  $1.25\mathrm{dB}$ . This evidence highlights the successful implementation of the pixel-level basis 3D LUTs fusion strategy discussed in Sec. 2.2. Next, we replace the regular lightweight CNN backbone with a tiny transformer backbone proposed by [23], which contains less than  $400\mathrm{K}$  parameters. After deploying the transformer backbone, the model is improved by  $0.93\mathrm{dB}$ , suggesting that the transformer backbone is more in line with global tone manipulation and benefits generating more visual pleasure LDR images. Furthermore, when employing the image-adaptive learnable local Laplacian filter, the results exhibit a significant improvement of  $1.28\mathrm{dB}$ . This finding indicates that the image-adaptive learnable local Laplacian filter facilitates the production of more vibrant results. As can be seen from Fig 6, combining local Laplacian filter with 3D LUT achieves good visual quality on both global and local enhancement in this challenging case. These results convincingly demonstrate the superiority of our proposed framework in tone mapping tasks.
Selection of the pyramid layers. We validate the influence of the number of Laplacian pyramid layers in this section. Our approach employs an adaptive Laplacian pyramid, allowing us to manipulate the
(b) + Weight map
Figure 6: Visual results of ablation study on framework component. (a) is the baseline method 3D LUT. (b) is apply the pixel-level weight map. (c) is deploy the transformer backbone. (d) is utilizing the learnable local Laplacian filter.
(c) + Transformer
(d) + Learnable Filter
number of layers by altering the resolution of the low-frequency image  $I_{low}$ . As shown in Tab. 3, the model performs best on all evaluation metrics when the resolution of  $I_{low}$  is set to  $256 \times 256$ . However, the proposed framework requires more computation. A trade-off between computational load and performance is determined by the number of layers in the Laplacian pyramid. The proposed framework remains robust when the resolution is reduced to alleviate the computational burden. For example, reducing the resolution of  $I_{low}$  from  $256 \times 256$  to  $64 \times 64$  only marginally decreases the PSNR of the proposed framework from 26.81 to 26.62. Remarkably, this reduction in resolution leads to a significant  $30\%$  decrease in computational burden. These results validate that the tone attributes are presented in a relatively low-dimensional space.
This paper proposes an effective end-to-end framework for HDR image tone mapping tasks, combining global and local enhancements. The proposed framework utilizes the Laplacian pyramid decomposition technique to handle high-resolution HDR images effectively. This approach significantly reduces computational complexity while simultaneously ensuring uncompromised enhancement performance. Global tone manipulation is performed on the low-frequency image using 3D LUTs. An image-adaptive learnable local Laplacian filter is proposed to progressively refine the high-frequency components, preserving local edge details and reconstructing the pyramids. Extensive experimental results on two publically available benchmark datasets show that our model performs favorably against state-of-the-art methods for both 480p and 4K resolutions.
Acknowledgements. This work was supported by the National Natural Science Foundation of China No.62176097, Hubei Provincial Natural Science Foundation of China No.2022CFA055.
[1] Mathieu Aubry, Sylvain Paris, Samuel W Hasinoff, Jan Kautz, and Frédo Durand. Fast local laplacian filters: Theory and applications. ACM Transactions on Graphics (TOG), 33(5):1-14, 2014. 2, 3, 5
[2] Francesco Banterle, Alessandro Artusi, Kurt Debattista, and Alan Chalmers. Advanced high dynamic range imaging. CRC press, 2017. 1
[3] Peter J Burt and Edward H Adelson. The laplacian pyramid as a compact image code. In Readings in computer vision, pages 671-679. Elsevier, 1987. 2, 3, 5
[4] Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. Learning photographic global tonal adjustment with a database of input/output image pairs. In CVPR 2011, pages 97-104. IEEE, 2011. 7, 9
[5] Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang. Deep photo enhancer: Unpaired learning for image enhancement from photographs with gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6306-6314, 2018. 2
[6] Paul Debevec and Simon Gibson. A tone mapping algorithm for high contrast images. In 13th eurographics workshop on rendering: Pisa, Italy. Citeseer, 2002. 2