SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kyusu Ahn $^{1,3}$  Byeonghyun Ko $^{2}$  HyunGyu Lee $^{2}$  Chanwoo Park $^{2}$  Jaejin Lee $^{1,2}$
Dept. of Data Science, Seoul National University, Seoul, Republic of Korea
$^{2}$ Dept. of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea
<sup>3</sup>Research Center, Samsung Display Co., Ltd., Yongin, Republic of Korea
kyusu.ahn@snu.ac.kr, {byeonghyun, hyungyu} @aces.snu.ac.kr, {99chanwoo, jaejin} @snu.ac.kr
Under Display Camera (UDC) is a novel imaging system that mounts a digital camera lens beneath a display panel with the panel covering the camera. However, the display panel causes severe degradation to captured images, such as low transmittance, blur, noise, and flare. The restoration of UDC-degraded images is challenging because of the unique luminance and diverse patterns of flares. Existing UDC dataset studies focus on unrealistic or synthetic UDC degradation rather than real-world UDC images. In this paper, we propose a real-world UDC dataset called UDC-SIT. To obtain the non-degraded and UDC-degraded images for the same scene, we propose an image-capturing system and an image alignment technique that exploits discrete Fourier transform (DFT) to align a pair of captured images. UDC-SIT also includes comprehensive annotations missing from other UDC datasets, such as light source, day/night, indoor/outdoor, and flare components (e.g., shimmers, streaks, and glares). We compare UDC-SIT with four existing representative UDC datasets and present the problems with existing UDC datasets. To show UDC-SIT's effectiveness, we compare UDC-SIT and a representative synthetic UDC dataset using five representative learnable image restoration models. The result indicates that the models trained with the synthetic UDC dataset are impractical because the synthetic UDC dataset does not reflect the actual characteristics of UDC-degraded images. UDC-SIT can enable further exploration in the UDC image restoration area and provide better insights into the problem. UDC-SIT is available at: https://github.com/mcrl/UDC-SIT.
Under Display Camera (UDC) is a technology designed to place a camera module under the display to use the UDC area as a display space and take pictures when the camera operates. Since a larger screen-to-body ratio is a common consumer demand that leads to a trend toward bezel-less display products [21, 26, 41], such as smartphones, laptops, tablets, and TVs, the display products equipped with UDC will well meet such a market trend. Moreover, video conferencing uses UDC products that enable natural eye-tracking by arranging a camera in the center of the display. However, UDC has some drawbacks of image deterioration problems, such as low transmittance, blur, noise, and flare.
Since the pixels in the micrometer scale diffract the light traveling through the camera lens [35], degrading image quality, UDC displays have a lower pixel density above the camera lens to reduce transmittance loss and diffraction, as shown in Figure 1. However, a lower display resolution in the UDC area affects natural video viewing. Thus, improving the quality of UDC images is a critical problem. Especially UDC image reconstruction models to enable higher pixel densities around the camera are essential to overcome the problem for more comfortable viewing.
Figure 1: Comparison of conventional hole-display and under-display cameras. The UDC area has a lower pixel density since the pixel pattern functions as diffraction slits. (a) Under Display Camera (UDC). (b) Hole display camera. (c) Pixel structure of the UDC area. (d) Comparison between the UDC area and other display area.
Several studies address the UDC image restoration problem. However, most of them [15, 50, 52] have limitations because their datasets do not completely reflect the properties of actual UDC images (i.e., are synthesized). A challenging problem in constructing the UDC dataset is finding a matching pair of the ground truth and distorted UDC images, which requires significant time and effort.
Feng et al. [13] proposes a pseudo-real-world UDC dataset. However, their dataset contains images with occlusions. Ignatov et al. [23, 24] try to match two images for the same scene captured by different devices with different positions and angles. They use the SIFT key points [32] of paired images and the RANSAC algorithm [16] to align the two images in a pair. However, the geometric alignment algorithm fails to adequately perform when there is a large discrepancy in the scenes of the two images, such as UDC degradation.
(a)
Figure 2: The image capturing system to capture a pair of matching images (the ground-truth image and the degraded image by the UDC). (a) The lid opens to get a ground-truth image. (b) The lid closes to get the corresponding degraded image.
This paper proposes a new UDC dataset called UDC-SIT (UDC's Still Images by the Thunder Research Group). As far as we know, it is the first real-world UDC dataset to overcome the problems of the existing UDC datasets.
We devise an image-capturing system that minimizes lens movement when capturing matching image pairs. We cut and take the UDC area of the flexible display panel of a smartphone with a UDC (e.g., Samsung Galaxy Z-Fold 3 [39]). We use the piece of the display panel as a lid on the camera lens of another smartphone with a non-UDC (e.g., Samsung Galaxy Note 10 [38]). The lid on the non-UDC can be opened and closed. As a result, our image-capturing system has minor geometric inconsistency compared to Feng et al. [13]. In addition, it helps to restrict rotations and tilts by the movement of the lid to a degree that does not notably influence alignment. Figure 2 illustrates our image-capturing system. We get a matching pair by capturing the ground-truth image with the lid off and the distorted image with the lid on.
We compensate for the pixel-position difference between the two images for the same scene by exploiting the discrete Fourier transform (DFT) [6]. The discrepancy in pixel positions is inevitable because of the slight movement of the camera when opening and closing the lid.
- We propose an image-capturing system for obtaining matched pairs of undistorted and UDC-distorted images for the same scene.
- We propose an image aligning technique for the paired images of the same scene by exploiting the discrete Fourier transform.
- We provide a real-world UDC dataset, UDC-SIT, which accurately reflects actual image degradations by the UDC. UDC-SIT offers a rich set of annotations that stimulate research in UDC image restoration. We make UDC-SIT publicly available (https://github.com/mcrl/UDC-SIT).
- To show UDC-SIT's effectiveness, we compare UDC-SIT and a representative synthetic UDC dataset using five representative learnable image restoration models.
Image restoration. Image restoration techniques restore a high-quality image from a degraded one. They include denoising, deblurring, and flare removal tasks. Camera-captured images often contain noisy and blurred pixels due to focus errors and incorrect light sensitivity. Restoring these images by removing noise and blur is a common task. Uformer [47] is one of the leading hierarchical transformer-based models for restoring such images. SIDD [1] comprises 30,000 noisy images captured by five smartphone cameras in 10 different scenes under various lighting conditions. DND [34] offers 50 pairs of noisy and noiseless images of very high resolution.
Light flares degrade image quality significantly due to strong light intensities. Wu et al. [48] generate synthetic flare removal datasets and train a neural network by modeling the optical characteristics of flares. Also, Dai et al. [11, 12] introduce a nighttime flare removal dataset to address the limitations of existing methods that only work well on daytime flares.
The UDC image restoration is complex, and its degradation patterns differ from other restoration tasks of images captured by standard cameras.
Existing UDC datasets. Zhou et al. [52] tackle the UDC image restoration problem using paired images from a Monitor Camera Imaging System (MCIS) and synthesized PSFs using optical modeling. However, their dataset has limitations, including unrealistic flares captured from a monitor with a limited dynamic range and inaccurate PSFs. They provide only 300 pairs of images for T-OLED and P-OLED, respectively.
Feng et al. [15] improve the UDC dataset. To measure the PSF, they place a white point light source one meter away from the OLED display of ZTE Axon 20 [10] and following the methodology presented by Sun et al. [43]. By convolving the PSF with HDR images from the HDRI Haven dataset [18], a synthetic dataset is generated. However, it lacks real-world characteristics because the images and PSFs come from different devices. Also, the flare shapes are limited, not including the distorted flares addressed by Yoo et al. [50].
Yoo et al. [50] introduce a synthetic UDC dataset that includes spatially varying PSFs obtained by optical simulation using the Brown-Conrady Distortion model [46]. However, their simulated flare distortions differ from real-world distortions, and their dataset is not publicly available.
Feng et al. [13] create a pseudo-real dataset by capturing degraded images with ZTE Axon 20 UDC [10] and ground-truth images with iPhone 13 Pro camera [25]. They face domain discrepancy and geometric misalignment challenges. Geometric misalignment is severe due to using different camera modules for the two paired images. AlignFormer [13] overcomes the geometric misalignment in UDC by aligning domain information with StyleConv [28] and AdaIN [22], and geometric information with an attention mechanism and a pre-trained optical flow estimator called RAFT [45].
DNN models for UDC image restoration. The UDC image restoration challenges [14, 53] use the datasets by Feng et al. [15] and Zhou et al. [52]. In these challenges, most of the top-ranked teams [31, 44, 49, 54] use the U-Net model [37] and residual networks [19] as their backbone Deep Neural Network (DNN) model for restoration.
3 Obtaining aligned images
This section describes our alignment technique of the standard and UDC images for the same scene. Misalignment between the two images is not due to the UDC itself but rather a problem that arises when capturing them. Previous techniques, such as SIFT [32] and RANSAC [16], are inadequate for this purpose because of severe degradation by the UDC. Feng et al. [13] use two cameras with
different specifications, leading to variations in perspectives and contents between paired images. Some tools, such as AlignFormer [13], align these images but introduce occlusion regions.
In contrast, our approach minimizes misalignment without introducing occlusion regions. After capturing the two images with our image-capturing system, we exploit discrete Fourier transform (DFT) to align the two paired images. We especially exploit their spatial frequency domain after the DFT to achieve degradation-resilient alignment [9, 17, 27].
3.1 Discrete Fourier transform
DFT converts a discrete signal represented by complex exponential waves into constituent frequencies. Equation 1 defines 2D DFT used to obtain the frequency representation of an image.
$$
\mathcal {F} (u, v) = \sum_ {x = 0} ^ {M - 1} \sum_ {y = 0} ^ {N - 1} f (x, y) \cdot e ^ {- i 2 \pi \left(\frac {u x}{M} + \frac {v y}{N}\right)}, \tag {1}
$$
where  $\mathcal{F}(u,v)$  denotes the frequency value,  $(u,v)$  represents a point in the frequency domain,  $M\times N$  is the image size, and  $f(x,y)$  is the pixel value at a point  $(x,y)$  in the image (i.e., in the spatial domain). Using Euler's formula decomposes the exponential function in Equation 1 into cosine and sine functions, and Equation 1 becomes
$$
\mathcal {F} (u, v) = \mathcal {R} (u, v) + i \mathcal {I} (u, v), \tag {2}
$$
where  $\mathcal{R}(u,v)$  and  $\mathcal{I}(u,v)$  denote the real and the imaginary part of  $\mathcal{F}(u,v)$ , respectively. The amplitude  $|\mathcal{F}(u,v)|$  and phase  $\phi(u,v)$  of  $\mathcal{F}(u,v)$  are defined as:
$$
\left| \mathcal {F} (u, v) \right| = \left[ \mathcal {R} ^ {2} (u, v) + \mathcal {I} ^ {2} (u, v) \right] ^ {\frac {1}{2}} \quad \text {a n d} \quad \phi (u, v) = \tan^ {- 1} \left[ \frac {\mathcal {I} (u , v)}{\mathcal {R} (u , v)} \right]. \tag {3}
$$
3.2 Alignment of paired images
Let  $M \times N$  be the image size. To measure the difference between the degraded image (D) and ground-truth image (G), we typically use Mean Squared Error (MSE),
$$
M S E = \sum_ {x = 0} ^ {M - 1} \sum_ {y = 0} ^ {N - 1} (D (x, y) - G (x, y)) ^ {2}. \tag {4}
$$
However,  $MSE$  primarily emphasizes local information at points  $(x,y)$ . In contrast, the spectrum's value for a specific point  $(u,v)$  in the frequency domain relies on the collective contribution of all points  $(x,y)$  in the spatial domain because  $\mathcal{F}(u,v)$  is the sum that iterates through each pixel  $(x,y)$  of the image in Equation 1. Thus, to align the two images, we assess the distance between paired images in both spatial and frequency domains as shown in Figure 3. To incorporate both local and global information, we employ a loss function that combines information in the spatial and frequency domains:
$$
L o s s = \lambda_ {1} \sum_ {x = 0} ^ {M - 1} \sum_ {y = 0} ^ {N - 1} (D (x, y) - G (x, y)) ^ {2} + \lambda_ {2} \sum_ {u = 0} ^ {M - 1} \sum_ {v = 0} ^ {N - 1} \Delta \mathcal {F} _ {a m p} (u, v) + \lambda_ {3} \sum_ {u = 0} ^ {M - 1} \sum_ {v = 0} ^ {N - 1} \Delta \phi (u, v), \tag {5}
$$
where  $\Delta \mathcal{F}_{amp}(u,v)$  is the L1 distance for the amplitude defined as  $\Delta \mathcal{F}_{amp}(u,v) = |\mathcal{F}_D(u,v) - \mathcal{F}_G(u,v)|$ , and  $\Delta \phi (u,v)$  is the L1 distance for the phase defined as  $\Delta \phi (u,v) = |\phi_D(u,v) - \phi_G(u,v)|$ .
$\mathcal{F}(u,v)$  at a point  $(u,v)$  represents a distinct spatial frequency component, and applying inverse DFT to  $\mathcal{F}(u,v)$  at a point  $(u,v)$  generates a sinusoidal grating in the spatial domain. Figure 4(a) and Figure 4(c) illustrate the isolation of sinusoidal gratings associated with  $\mathcal{F}_G(u,v)$  and  $\mathcal{F}_D(u,v)$ , respectively.
The significant advantage of using DFT in aligning the original and shifted images lies in its capability to decompose an image into its constituent spatial frequency components. Figure 4(a) and Figure 4(c) display the initial three low-frequency sinusoidal gratings from the two images, clearly revealing a distinct spatial shift. Notably, while the amplitude of sinusoidal gratings remains constant within the same column of Figure 4(a) and Figure 4(c), their corresponding phases exhibit variations. Figure 4(b)
Figure 3: Overview of the proposed loss function to align two paired images. The loss function considers the difference between degraded (D) and ground-truth (G) images in both the spatial and frequency domains.
Figure 4: Frequency analysis of two paired images. One is the original image (G), and the other is the spatially shifted image (D) of G. Let  $\mathcal{F}_X(u,v)$  be the result of applying  $DFT$  to an image  $X$ . (a) G in the spatial domain comprises multiple sinusoidal gratings. Each sinusoidal grating results from applying inverse  $DFT$  to  $\mathcal{F}_G(u,v)$ . (b) The amplitude difference between G and D. There is no difference. (c) Similar to G, D in the spatial domain comprises multiple sinusoidal gratings. (d) The phase difference between G and D.
and Figure 4(d) visually depict the differences in amplitude (Figure 4(b)) and phase (Figure 4(d)) between the two images. Reducing  $\Delta \phi$  is crucial for effectively aligning a misaligned pair of images for the same scene. Figure 4 is a conceptual illustration for paired images involving shifts without degradation. However, due to the degradation, UDC-degraded images also exhibit differences from the ground truth in amplitude. Figure 4 elucidates the importance of the phase component in the alignment of the images.
Our alignment algorithm minimizes the loss in Equation 5 between the ground-truth image and the degraded image. The misaligned degraded image is rotated, shifted and then cropped to achieve the same size as the ground truth until the loss reaches the minimum. Handling rotation and shifts are manageable, but addressing tilt presents a challenge. It is because tilt requires perspective transforms optimized for objects within a single image sharing the same plane. Thus, our current emphasis is on tackling shifts and rotations, with tilt considerations excluded. Nonetheless, our data collection is meticulous, limiting rotations and tilts to a degree that does not notably influence the alignment, as the PCK values in Table 2 affirm.
The original size of camera-captured images is (2016, 1512, 4). The ground truth image is center-cropped to (1792, 1280, 4). The degraded image is similarly cropped around the center. For the degraded image, iterative shifting of  $(\mathbf{x},\mathbf{y})$  coordinates and rotation are used to find the minimum loss point where the cropped degraded image aligns with the cropped ground truth image. The final cropped image size becomes (1792, 1280, 4) to ensure H and W are multiples of 256. The detailed algorithm is illustrated in Algorithm 1. Here, we establish  $s = 20$ ,  $\theta = 0.3$ ,  $r = 0.1$ , and  $(\lambda_1,\lambda_2,\lambda_3)$  combinations in Table 3 as hyperparameters.
Algorithm 1 Alignment of paired images  $I_{G}$  and  $I_{D}$
Require: Images  $I_{G}, I_{D}$  of size  $(H, W)$ , hyperparameters  $s, \theta_r, r, \lambda_1, \lambda_2, \lambda_3$   
Ensure: Aligned images  $C_{G}, C_{D}$  of size  $(H^{*}, W^{*})$   
Crop  $C_{G}$  from  $I_{G}$  using center crop  
Crop  $C_{D}$  from  $I_{D}$  to the size of  $C_{G}$   
Initialize best loss  $L_{\text{best}}$  to a large value  
Initialize optimal shifts  $s_{\text{opt\_x}}, s_{\text{opt\_y}}$ , and rotation  $\theta_{\text{opt}}$  to 0  
for  $\theta_{\text{rotation}}$  from  $-\theta_r$  to  $\theta_r$  with step  $r$  do  
Apply rotation of  $\theta_{\text{rotation}}$  to  $I_{D}$  to get  $I_{D_{\text{rotated}}}$   
for  $x_{\text{shift}}$  from  $-s$  to  $s$  with step 1 do  
for  $y_{\text{shift}}$  from  $-s$  to  $s$  with step 1 do  
Calculate crop position  $(p, q)$  relative to the center crop:  
 $p = x_{\text{center\_crop}} + x_{\text{shift}}$ $q = y_{\text{center\_crop}} + y_{\text{shift}}$   
Crop image  $C_{D_{\text{tmp}}}^{\text{tmp}}$  from  $I_{D_{\text{rotated}}}$  at position  $(p, q)$   
Calculate loss  $L$  using the loss function in Eq. 5 between  $C_{D_{\text{tmp}}}^{\text{tmp}}$  and  $C_{G}$   
if  $L < L_{\text{best}}$  then  
Update  $L_{\text{best}}$  to  $L$   
Update  $s_{\text{opt\_x}}$  to  $x_{\text{shift}}$   
Update  $s_{\text{opt\_y}}$  to  $y_{\text{shift}}$   
Update  $\theta_{\text{opt}}$  to  $\theta_{\text{rotation}}$   
end if  
end for  
end for  
end for  
Apply optimal rotation  $\theta_{\text{opt}}$  to  $I_{D}$  to get  $I_{D_{\text{rotated}}}$   
Calculate crop position  $(p_{\text{opt}}, q_{\text{opt}})$  relative to the center crop:  
 $p_{\text{opt}} = x_{\text{center\_crop}} + s_{\text{opt\_x}}$ $q_{\text{opt}} = y_{\text{center\_crop}} + s_{\text{opt\_y}}$   
Crop  $I_{D_{\text{rotated}}}$  to acquire an aligned image  $C_{D}$  at position  $(p_{\text{opt}}, q_{\text{opt}})$
We capture images in UDC-SIT without any Image Signal Processing (ISP) and in RAW format. While High Dynamic Range (HDR) captures more details in shadows and highlights, authentic UDC images are typically in Low Dynamic Range (LDR). Generating HDR images requires capturing multiple LDR images with different exposures, then combining them to create an HDR image [2]. This process differs from general photography, so we gather our dataset in LDR.
Table 1: Comparison of the UDC datasets. Unlike others, UDC-SIT provides annotations, such as light source, day/night, indoor/outdoor, and flare types. Flare types are classified as shimmer, streak, and glare.
Dataset	Scene	Dynamic range	Dataset size	Annotations	Publicly available
Zhou et al. [52]	Synthetic	LDR	300	No	Yes
Feng et al. [15]	Synthetic	HDR	2,376	No	Yes
Yoo et al. [50]	Synthetic	LDR	-	No	No
Feng et al. [13]	Real	HDR	6,747	No	Yes
UDC-SIT	Real	LDR	2,340	Yes	Yes
4 Comparison with the existing UDC datasets
In this section, we compare UDC-SIT with the four existing representative UDC image datasets. We summarize the five datasets Zhou-S [52], Feng-S [15], Yoo-S [50], Feng-R [13], and UDC-SIT in Table 1, where S and R stand for synthetic and real datasets, respectively. Feng et al. capture 330 images and then crop them into 6,747 small patches. We explain the detail in the appendix.
Noises and transmittance decrease. Under low-light conditions, the camera sensor amplifies both the desired signal and unintended random noise. Since the camera sensor in the UDC is positioned beneath the display pixels that decrease the transmittance, the camera operates in low-light conditions, resulting in noise amplification. The degraded images in the UDC dataset should effectively contain the unique UDC noise, which differs from standard cameras. For example, a
(a) Zhou-S
(b) Feng-R
(c) Feng-S
(d) UDC-SIT
Figure 5: Comparison of the transmittance decrease and UDC noises. GT stands for ground truth. (a) Zhou-S [52]. (b) Feng-R [13]. (c) Feng-S [15]. (d) UDC-SIT.
(a) Zhou-S
(b) Feng-R
(c) Feng-S
(d) UDC-SIT
(e) Zhou-S
(f) Feng-R
(g) Feng-S
(h) UDC-SIT
Figure 6: Comparison of flares. The light sources affect the flare shape. The images in (a) Zhou-S [52], (b) Feng-R [13], (c) Feng-S [15], and (d) UDC-SIT show the UDC flares by the round shape of light sources. The images in (e) Zhou-S [52], (f) Feng-R [13], (g) Feng-S [15], and (h) UDC-SIT show the UDC flares by fluorescent light sources.
visual comparison is given in Figure 5. The image in Zhou-S in Figure 5(a) is captured from a monitor under controlled lighting conditions. Thus, there exists no noise. However, we observe the excessive transmittance decrease caused by P-OLED. In addition, the image in Feng-S in Figure 5(c) is generated by convolving the PSF on an HDR image from the HDRI Haven dataset [18]. The HDR images in the dataset are meticulously created through exposure bracketing [3] and cleanup to ensure high quality without overexposure, chromatic aberration, and noise. Thus, the Feng-S image has reduced noise and minimal transmittance loss. On the other hand, the images in Feng-R in Figure 5(b) and in UDC-SIT in Figure 5(d) accurately depict the visible noise caused by the UDC. They also show the effect of transmittance decrease.
Table 2: The PCK comparison between the datasets. UDC-SIT shows the best alignment and has PCK values close to  $100\%$  for all values of  $\alpha$ .
Dataset	Need alignment	PCK (α = 0.01)	PCK (α = 0.03)	PCK (α = 0.10)
Zhou-S [52]	98.11	98.45	99.08
Feng-S [15]	99.95	99.96	99.99
Feng-R [13]	✓	58.75	95.08	99.93
UDC-SIT	✓	97.26	98.56	99.35
Flares. Lens flares typically occur when intense light scatters or reflects within an optical system [11]. In contrast, UDC flares result from light interacting with the display panel above the camera sensor, resulting in undesired reflections, diffraction, or scattering. Thus, it is essential for the images in the UDC dataset to exactly describe real UDC flares if they exist. For example, a visual comparison is given in Figure 6. The images in Zhou-S in Figure 6(a) and (e) do not exhibit any flares since they capture images displayed on a monitor. Similarly, the images in Feng-S in Figure 6(c) and (g) generate unrealistic flares by convolving PSFs to HDR images. Specifically, it lacks the generation of glare, streaks, and spatially variant flares. The shapes of real-world UDC flares vary depending on their positions, resulting in different diffraction patterns. However, the flares in the image in Feng-S in Figure 6(c) appear overly regular. (Figure 6(c) versus (b) and (d)). Moreover, the image in Feng-S in Figure 6(g) demonstrates different fluorescent light flares from the real-world UDC flares in Figure 6(f) and (h). Figure 6(b) versus (d) and (f) versus (h) depict various flare shapes from different UDC smartphones.