SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Feng Zhang $^{1}$  Ming Tian $^{1}$  Zhiqiang Li $^{2}$  Bin Xu $^{2}$  Qingbo Lu $^{2}$  Changxin Gao $^{1,3*}$  Nong Sang $^{1}$
$^{1}$  National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology,
$^{2}$  DJI Technology Co., Ltd,  $^{3}$  Hubei Key Laboratory of Brain-inspired Intelligent Systems {fengzhangaia, tianming, cgao, nsang}@hust.edu.cn, {mila.xu, cristopher.li, qingbo.lu}@dji.com
Tone mapping aims to convert high dynamic range (HDR) images to low dynamic range (LDR) representations, a critical task in the camera imaging pipeline. In recent years, 3-Dimensional Look-Up Table (3D LUT) based methods have gained attention due to their ability to strike a favorable balance between enhancement performance and computational efficiency. However, these methods often fail to deliver satisfactory results in local areas since the look-up table is a global operator for tone mapping, which works based on pixel values and fails to incorporate crucial local information. To this end, this paper aims to address this issue by exploring a novel strategy that integrates global and local operators by utilizing closed-form Laplacian pyramid decomposition and reconstruction. Specifically, we employ image-adaptive 3D LUTs to manipulate the tone in the low-frequency image by leveraging the specific characteristics of the frequency information. Furthermore, we utilize local Laplacian filters to refine the edge details in the high-frequency components in an adaptive manner. Local Laplacian filters are widely used to preserve edge details in photographs, but their conventional usage involves manual tuning and fixed implementation within camera imaging pipelines or photo editing tools. We propose to learn parameter value maps progressively for local Laplacian filters from annotated data using a lightweight network. Our model achieves simultaneous global tone manipulation and local edge detail preservation in an end-to-end manner. Extensive experimental results on two benchmark datasets demonstrate that the proposed method performs favorably against state-of-the-art methods.
Modern cameras, despite their advanced and sophisticated sensors, are limited in their ability to capture the same level of detail as the human eye in a given scene. In order to capture more detail, high dynamic range (HDR) imaging techniques [25, 2] have been developed to convey a wider range of contrasts and luminance values than conventional low dynamic range (LDR) imaging. However, most modern graphics display devices have a limited dynamic range that is inadequate to reproduce the full range of light intensities present in natural scenes. To tackle this issue, tone mapping techniques [28, 34, 9] have been proposed to render high-contrast scene radiance to the
*Changxin Gao is the corresponding author, email: cgao@hust.edu.cn
displayable range while preserving the image details and color appearance important to appreciate the original scene content.
Traditional tone mapping operators can be classified according to their processing as global or local. Global operators [11, 34, 7, 33, 21] map each pixel according to its global characteristics, irrespective of its spatial localization. This approach entails calculating a single matching luminance value for the entire image. As a result, the processing time is considerably reduced, but the resulting image may exhibit fewer details. In contrast, local operators [6, 8, 10, 24, 29] consider the spatial localization of each pixel within the image and process them accordingly. In essence, this method calculates the luminance adaptation for each pixel based on its specific position. Consequently, the resulting image becomes more visually accessible to the human eye and exhibits enhanced details, albeit at the expense of longer processing times. However, these traditional operators often require manual tuning by experienced engineers, which can be cumbersome since evaluating results necessitates testing across various scenes. Although system contributions have aimed to simplify the implementation of high-performance executables [32, 15, 27], they still necessitate programming expertise, incur runtime costs that escalate with pipeline complexity, and are only applicable when the source code for the filters is available. Therefore, seeking an automatic strategy for HDR image tone mapping is of great interest.
In recent years, there have been notable advancements in learning-based automatic enhancement methods [37, 12, 5, 30, 16, 35, 19], thanks to the rapid development of deep learning techniques [22]. Many of these methods focus on learning a dense pixel-to-pixel mapping between input high dynamic range (HDR) and output low dynamic range (LDR) image pairs. Alternatively, they predict pixel-wise transformations to map the input HDR image. However, most previous studies involve a substantial computational burden that exhibits a linear growth pattern in tandem with the dimensions of the input image.
To simultaneously improve the quality and efficiency of learning-based methods, hybrid methods [17, 41, 38, 36, 39] have emerged that combine the utilization of image priors from traditional operators with the integration of multi-level features within deep learning-based frameworks, leading to state-of-the-art performance. Among these methods, Zeng et al. [38] proposes a novel image-adaptive 3-Dimensional Look-Up Table (3D LUT) based approach, which exhibits favorable characteristics such as superior image quality, efficient computational processing, and minimal memory utilization. However, as indicated by the authors, utilizing the global (spatially uniform) tone mapping operators, such as the 3D look-up tables, may produce less satisfactory results in local areas. Additionally, this method necessitates an initial downsampling step to reduce network computations. In the case of high-resolution (4K) images, this downsampling process entails a substantial reduction factor of up to 16 times (typically downsampled to  $256 \times 256$  resolution). Consequently, this results in a significant loss of image details and subsequent degradation in enhancement performance.
To alleviate the above problems, this work focuses on integrating global and local operators to facilitate comprehensive tone mapping. Drawing inspiration from the reversible Laplacian pyramid decomposition [3] and the classical local tone mapping operators, the local Laplacian filter [29, 1], we propose an effective end-to-end framework for the HDR image tone mapping task performing global tone manipulation while preserving local edge details. Specifically, we build a lightweight transformer weight predictor on the bottom of the Laplacian pyramid to predict the pixel-level content-dependent weight maps. The input HDR image is trilinear interpolated using the basis 3D LUTs and then multiplied with weighted maps to generate a coarse LDR image. To preserve local edge details and reconstruct the image from the Laplacian pyramid faithfully, we propose an image-adaptive learnable local Laplacian filter (LLF) to refine the high-frequency components while minimizing the use of computationally expensive convolution in the high-resolution components for efficiency. Consequently, we progressively construct a compact network to learn the parameter value maps at each level of the Laplacian pyramid and apply them to the remapping function of the local Laplacian filter. Moreover, a fast local Laplacian filter [1] is employed to replace the conventional local Laplacian filter [29] for computational efficiency. Extensive experimental results on two benchmark datasets demonstrate that the proposed method performs favorably against state-of-the-art methods.
In conclusion, the highlights of this work can be summarized into three points:
(1) We introduce an effective end-to-end framework for HDR image tone mapping. The network performs both global tone manipulation and local edge details preservation within the same model.
Figure 1: Overview of the framework. Our method first decomposes the input image  $\mathbf{I}$  into a Laplacian pyramid. The low-frequency image  $\mathbf{I}_{low}$  is fed into a lightweight transformer weight predictor and Basis 3D LUTs fusion block to transform into a low-resolution enhanced image  $\hat{\mathbf{I}}_{low}$ . To adaptively refine the high-frequency components, we progressively learn an image-adaptive local Laplacian filter (LLF) based on both high- and low-frequency images. Then, we perform the remapping function of the local Laplacian filter to refine the high-frequency components while preserving the pyramid reconstruction capability. For the level  $N - 1$ , we concatenate the component with the edge map of  $\hat{\mathbf{I}}_{low}$  to mitigate potential halo artifacts.
(2) We propose an image-adaptive learnable local Laplacian filter for efficient local edge details preservation, demonstrating remarkable effectiveness when integrated with image-adaptive 3D LUT.
(3) We conduct extensively experiments on two publically available benchmark datasets. Both qualitative and quantitative results demonstrate that the proposed method performs favorably against state-of-the-art methods.
We propose an end-to-end framework to manipulate tone while preserving local edge detail in HDR image tone mapping tasks. The pipeline of our proposed method is illustrated in Fig. 1. Given an input 16-bit HDR image  $\mathbf{I} \in \mathbb{R}^{h \times w \times 3}$ , we initially decompose it into an adaptive Laplacian pyramid, resulting in a collection of high-frequency components represented by  $\mathbf{L} = [l_0, l_1, \dots, l_{N-1}]$ , as well as a low-frequency image denoted as  $\mathbf{I}_{low}$ . Here,  $N$  represents the number of decomposition levels in the Laplacian pyramid. The adaptive Laplacian pyramid employs a dynamic adjustment of the decomposition levels to match the resolution of the input image. This adaptive process ensures that the low-frequency image  $\mathbf{I}_{low}$  achieves a proximity of approximately  $64 \times 64$  resolution. The described decomposition process possesses invertibility, allowing the original image to be reconstructed by incremental operations. According to Burt and Adelson [3], each pixel in the low-frequency image  $\mathbf{I}_{low}$  is averaged over adjacent pixels by means of an octave Gaussian filter, which reflects the global characteristics of the input HDR image, including color and illumination attributes. Meanwhile, other high-frequency components contain edge-detailed textures of the image.
Motivated by the characteristics mentioned above of the Laplacian pyramid, we propose to manipulate tone on  $\mathbf{I}_{low}$  while refining the high-frequency components  $\mathbf{L}$  progressively to preserve local edge details. In addition, we progressively refine the higher-resolution component conditioned on the lower-resolution one. The proposed framework consists of three parts. Firstly, we introduce a lightweight transformer block to process  $\mathbf{I}_{low}$  and generate content-dependent weight maps. These predicted weight maps are employed to fuse the basis of 3D LUTs. Subsequently, this adapted representation is used to transform  $\mathbf{I}_{low}$  into  $\hat{\mathbf{I}}_{low}$ , resulting in the desired tone manipulation. Secondly, we construct parameter value maps by leveraging a learned model on the concatenation of  $[l_{N-1}, up(\mathbf{I}_{low}), up(edge(\hat{\mathbf{I}}_{low}))]$ , where  $up(\cdots)$  represents a bilinear up-sampling operation and  $edge(\cdots)$  denotes for the canny edge detector. These parameter value maps are then employed to perform a fast local Laplacian filter [1] on the Laplacian layer of level  $N-1$ . This step effectively
(c)
Figure 2: Illustration of the basis 3D LUTs fusion strategy. (a) present the multiple pixel mapping relationships of an image pair; (b) is the conventional basis 3D LUTs fusion strategy; (c) is the pixel-level basis 3D LUTs fusion strategy.
refines the high-frequency components while considering the local edge detail information. Lastly, we propose an efficient and progressive upsampling strategy to further enhance the refinement of the remaining Laplacian layers with higher resolutions. Starting from level  $l = N - 2$  down to  $l = 0$ , we sequentially upsample the refined components from the previous level and concatenate them with the corresponding Laplacian layer. Subsequently, we employ a lightweight convolution block to perform another fast local Laplacian filter. This iterative process iterates across multiple levels, effectively refining the high-resolution components. We introduce these modules in detail in the following sections.
2.2 Pixel-level Basis 3D LUTs Fusion
According to the inherent properties of the Laplacian pyramid, the low-frequency image contains properties such as color and illumination of the images. Therefore, we employ 3D LUTs to perform tone manipulation on low-frequency images  $\mathbf{I}_{low}$ . In RGB color space, a 3D LUT defines a 3D lattice that consists of  $N_{b}^{3}$  elements, where  $N_{b}$  is the number of bins in each color channel. Each element defines a pixel-to-pixel mapping function  $\mathbf{M}^c (i,j,k)$ , where  $i,j,k = 0,1,\dots ,n_b - 1\in \mathbb{I}_0^{N_b - 1}$  are elements' coordinates within 3D lattice and  $c$  indicates color channel. Given an input RGB color  $\{(\mathbf{I}_{(i,j,k)}^r,\mathbf{I}_{(i,j,k)}^g,\mathbf{I}_{(i,j,k)}^b)\}$ , where  $i,j,k$  are indexed by the corresponding RGB value, a output  $\mathbf{O}^c$  is derived by the mapping function as follows:
$$
\mathbf {O} _ {(i, j, k)} ^ {c} = \mathbf {M} ^ {c} \left(\mathbf {I} _ {(i, j, k)} ^ {r}, \mathbf {I} _ {(i, j, k)} ^ {g}, \mathbf {I} _ {(i, j, k)} ^ {b}\right). \tag {1}
$$
The mapping capabilities of conventional 3D LUTs are inherently constrained to fixed transformations of pixel values. Fig. 2(a) demonstrates this limitation, where the input image has the same pixel values at different locations. However, these locations contain different pixel values in the reference image. While the input image is interpolated through a look-up table, the transformed image retains the same transformed pixel values at these locations. Consequently, the conventional 3D LUT framework fails to accommodate intricate pixel mapping relationships, thus impeding its efficacy in accurately representing such pixel transformations.
Inspired by [36], we propose an effective 3D LUT fusion strategy to address this inherent limitation. The conventional 3D LUT fusion strategy proposed by [38] is shown in Fig. 2(b), which first utilizes the predicted weights to fuse the multiple 3D LUTs into an image-adaptive one and then performs trilinear interpolation to transform images. In contrast, as shown in Fig. 2(c), our strategy is first to perform trilinear interpolation with each LUT and then fuse the enhanced image with predicted pixel-level weight maps. In this way, our method can enable a relatively more comprehensive and accurate representation of the complex pixel mapping relationships through the weight values of each pixel. The pixel-level mapping function  $\Phi^{h,w,c}$  can be described as follows:
$$
\mathbf {O} _ {(i, j, k)} ^ {h, w, c} = \boldsymbol {\Phi} ^ {h, w, c} \left(\mathbf {I} _ {(i, j, k)} ^ {r}, \mathbf {I} _ {(i, j, k)} ^ {g}, \mathbf {I} _ {(i, j, k)} ^ {b}, \omega^ {h, w}\right) = \sum_ {n = 0} ^ {N - 1} \omega_ {n} ^ {h, w} \mathbf {M} _ {n} ^ {c} \left(\mathbf {I} _ {(i, j, k)} ^ {r}, \mathbf {I} _ {(i, j, k)} ^ {g}, \mathbf {I} _ {(i, j, k)} ^ {b}\right), \tag {2}
$$
where  $\mathbf{O}_{(i,j,k)}^{h,w,c}$  is the final pixel-level output,  $\omega_{n}^{h,w}$  represents a pixel-level weight map for  $N$  3D LUTs located at  $(h,w)$ . Note that our proposed strategy involves the utilization of multiple trilinear
interpolations, which may impact the computational speed when applied to high-resolution images. However, since our method operates at the resolution of  $64 \times 64$ , the computational overhead is insignificant. More discussions are provided in the supplementary material.
As shown in Fig. 1, given  $\mathbf{I}_{low}$  with a reduced resolution, we feed it into a weight predictor to output the content-dependent weight maps  $\omega^{h,w}$ . Since the weight predictor aims to understand the global context, such as the brightness, color, and tone of an image, a transformer backbone is more suitable for extracting global information than a CNN backbone. Therefore, we utilize a tiny transformer model proposed by [23] as the weight predictor. The whole model contains only  $400\mathrm{K}$  parameters when  $N = 3$ .
2.3 Image-adaptive Learnable Local Laplacian Filter
Although the pixel-level basis 3D LUTs fusion strategy demonstrates stable and efficient enhancement of input images across various scenes, the transformation of pixel values through weight maps alone still falls short of significantly improving local detail and contrast. To tackle this limitation, one potential solution is to integrate a local enhancement method with 3D LUT. In this regard, drawing inspiration from the intrinsic characteristics of the Laplacian pyramid [3], which involves texture separation, visual attribute separation, and reversible reconstruction, the combination of 3D LUT and the local Laplacian filter [29] can offer substantial benefits.
Figure 3: Architecture of the proposed image-adaptive learnable local Laplacian filter (LLF).
Local Laplacian filters are edge-aware local tone
mapping operators that define the output image by constructing its Laplacian pyramid coefficient by coefficient. The computation of each coefficient  $i$  is independent of the others. These coefficients are computed by the following remapping functions  $\mathbf{r}(\mathbf{i})$ :
$$
\mathbf {r} (\mathbf {i}) = \left\{ \begin{array}{l l} g + \operatorname {s i g n} (i - g) \sigma_ {r} \left(\left| i - g \right| / \sigma_ {r}\right) ^ {\alpha} & \text {i f} i \leq \sigma_ {r} \\ g + \operatorname {s i g n} (i - g) \left(\beta \left(\left| i - g \right| - \sigma_ {r}\right) + \sigma_ {r}\right) & \text {i f} i > \sigma_ {r} \end{array} , \right. \tag {3}
$$
where  $g$  is the coefficient of the Gaussian pyramid at each level, which acts as a reference value,  $\text{sign}(x) = x / |x|$  is a function that returns the sign of a real number,  $\alpha$  is one parameter that controls the amount of detail increase or decrease,  $\beta$  is another parameter that controls the dynamic range compression or expansion, and  $\sigma_r$  defines the intensity threshold the separates details from edges.
Nevertheless, the conventional approach described in Eq. 3 necessitates manual parameter adjustment for each input image, leading to a cumbersome and labor-intensive process. To overcome this limitation, we propose an image-adaptive learnable local Laplacian filter (LLF) to learn the parameter value maps for the remapping function. The objective function of the learning scheme can be written as follows:
$$
\min  _ {\alpha , \beta} \mathcal {L} (\mathbf {r} (l, g), \mathbf {R}), \tag {4}
$$
where  $\alpha$  and  $\beta$  are the learned parameter value maps of the Laplacian pyramid,  $\mathcal{L}(\dots)$  denotes the loss functions,  $\mathbf{r}(l,g)$  presents the image-adaptive learnable local Laplacian filter (LLF),  $l$  and  $g$  are the coefficients of the Laplacian and Gaussian pyramid, respectively,  $\mathbf{R}$  is the reference image. Note that the parameter  $\sigma_r$  does not impact the filter's performance; thus, it is fixed at 0.1 in this paper. Furthermore, to enhance computational efficiency, we have employed the fast local Laplacian filter [1] instead of the conventional local Laplacian filter.
As discussed in Sec. 2.1, we have  $l_{N - 1}\in \mathbb{R}^{\frac{h}{2N - 1}\times \frac{w}{2N - 1}\times 3}$  and  $\mathbf{I}_{low},\hat{\mathbf{I}}_{low}\in \mathbb{R}^{\frac{h}{2N}\times \frac{w}{2N}\times 3}$ . To address potential halo artifacts, we initially employ a Canny edge detector with default parameters to extract the edge map of  $\hat{\mathbf{I}}_{low}$ . Subsequently, we upsample  $\mathbf{I}_{low}$  and  $edge(\hat{\mathbf{I}}_{low})$  using bilinear operations to match the resolution of  $l_{N - 1}$  and concatenate them. The concatenated components are fed into a Parameter Prediction Block (PPB) as depicted in Fig. 3. The outputs of the PPB are utilized for the
Methods	#Params	HDR+ (480p)	HDR+ (original)
PSNR↑	SSIM↑	LPIPS↓	△E↓	PSNR↑	SSIM↑	LPIPS↓	△E↓
UPE [35]	999K	23.33	0.852	0.150	7.68	21.54	0.723	0.361	9.88
HDRNet [12]	482K	24.15	0.845	0.110	7.15	23.94	0.796	0.266	6.77
CSRNet [14]	37K	23.72	0.864	0.104	6.67	22.54	0.766	0.284	7.55
DeepLPF [26]	1.72M	25.73	0.902	0.073	6.05	N.A.	N.A.	N.A.	N.A.
LUT [38]	592K	23.29	0.855	0.117	7.16	21.78	0.772	0.303	9.45
sLUT [36]	4.52M	26.13	0.901	0.069	5.34	23.98	0.789	0.242	6.85
CLUT [39]	952K	26.05	0.892	0.088	5.57	24.04	0.789	0.245	6.78
Ours	731K	26.62	0.907	0.063	5.31	25.32	0.849	0.149	6.03
Table 1: Quantitative comparison on  $\mathrm{HDR + }$  [13] dataset. "N.A." means that the results are not available due to insufficient memory of the GPU.
remapping function  $\mathbf{r}(\mathbf{i})$  to refine  $l_{N - 1}$
$$
\hat {l} _ {N - 1} = \mathbf {r} \left(l _ {N - 1}, g _ {N - 1}, \alpha_ {N - 1}, \beta_ {N - 1}\right). \tag {5}
$$
Subsequently, we adopt a progressive upsampling strategy to match the refined high-frequency component  $\hat{l}_{N-1}$  with the remaining high-frequency components. This upsampled component is concatenated with  $l_{N-2}$ . As depicted in Fig. 1, the concatenated vector  $[l_{N-2}, up(\hat{l}_{N-1})]$  is feed into another LLF. The refinement process continues iteratively, progressively upsampling until  $\hat{l}_0$  is obtained. By applying the same operations as described in Eq. 5, all high-frequency components are effectively refined, leading to a set of refined components  $[\hat{l}_0, \hat{l}_1, \dots, \hat{l}_{N-1}]$ . Finally, the result image  $\hat{\mathbf{I}}$  is reconstructed using the tone mapped image  $\hat{\mathbf{I}}_{low}$  with refined components  $[\hat{l}_0, \hat{l}_1, \dots, \hat{l}_{N-1}]$ .
2.4 Overall Training Objective
The proposed framework is trained in a supervised scenario by optimizing a reconstruction loss. To encourage a faithful global and local enhancement, given a set of image pairs  $(\mathbf{I},\mathbf{R})$ , where  $\mathbf{I}[i]$  and  $\mathbf{R}[i]$  denote a pair of 16-bit input HDR and 8-bit reference LDR image, we define the reconstruction loss function as follows:
$$
\mathcal {L} _ {1} = \sum_ {i = 1} ^ {H \times W} \left(\| \hat {\mathbf {I}} [ i ] - \mathbf {R} [ i ] \| _ {1} + \| \hat {\mathbf {I}} _ {l o w} [ i ] - \mathbf {R} _ {l o w} [ i ] \| _ {1}\right), \tag {6}
$$
where  $\hat{\mathbf{I}}[i]$  is the output of network with  $\mathbf{I}[i]$  as input,  $\hat{\mathbf{I}}_{low}[i]$  is the output of 3D LUT with  $\mathbf{I}_{low}[i]$  as input,  $\mathbf{R}_{low}[i]$  is the low-frequency image of reference image  $\mathbf{R}[i]$ .
To make the learned 3D LUTs more stable and robust, some regularization terms from [38], including smoothness term  $\mathcal{L}_s$  and monotonicity term  $\mathcal{L}_m$ , are employed. In addition to these terms, we employ an LPIPS loss [40] function that assesses a solution concerning perceptually relevant characteristics (e.g., the structural contents and detailed textures):
$$
\mathcal {L} _ {p} = \sum_ {l} \frac {1}{H ^ {l} W ^ {l}} \sum_ {h, w} \| \phi (\hat {\mathbf {I}}) _ {h w} ^ {l} - \phi (\mathbf {R}) _ {h w} ^ {l} \| _ {2} ^ {2}, \tag {7}
$$
where  $\phi (\cdot)_{hw}^{l}$  denotes the feature map of layer  $l$  extracted from a pre-trained AlexNet [20].
To summarize, the complete objective of our proposed model is combined as follows:
$$
\mathcal {L} = \mathcal {L} _ {1} + \lambda_ {s} \mathcal {L} _ {s} + \lambda_ {m} \mathcal {L} _ {m} + \lambda_ {p} \mathcal {L} _ {p}, \tag {8}
$$
where  $\lambda_s, \lambda_m$ , and  $\lambda_p$  are hyper-parameters to control the balance of loss functions. In our experiment, these parameters are set to  $\lambda_s = 0.0001$ ,  $\lambda_m = 10$ ,  $\lambda_p = 0.01$ .