SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 7: Invalid mask visualization in Feng-R [13]. (a) The degraded image captured by the ZTE Axon 20 [10] UDC. (b) The ground-truth image captured by the iPhone 13 Pro rear camera [25]. (c) Aligned ground-truth image by AlignFormer [13].
Occlusion regions. Although Feng-R consists of real images, it has some limitations. They avoid capturing close objects to avoid parallax and occlusion. Nonetheless, AlignFormer [13] used in Feng-R creates occlusion regions marked red in Figure 7. This area is typically excluded in training the DNN restoration models. This is why they call Feng-R as pseudo-real pairs.
Table 3: Analyzing UDC-SIT's PCK relative to  $\lambda_{1}$ ,  $\lambda_{2}$ ,  $\lambda_{3}$ , and  $\theta_{\mathrm{rotation}}$ . When no rotation is applied,  $\theta_{\mathrm{rotation}} = 0$ ; otherwise,  $\theta_{\mathrm{rotation}} = \theta_r$ . Notably,  $\lambda_{2} = 1$  significantly improves the alignment over using MSE alone (i.e., the case of  $(1,0,0,0)$ ).
(λ1,λ2,λ3,θrotation)	PCK (α = 0.01)	PCK (α = 0.03)	PCK (α = 0.10)
(1,0,0,0)	78.77	81.09	85.65
(0,1,0,0)	34.05	50.70	64.66
(0,0,1,0)	62.27	64.59	72.54
(1,1,0,0)	35.55	49.41	63.25
(1,0,1,0)	98.22	98.73	99.31
(1,1,1,0)	86.13	95.02	99.23
(1,0.1,0.1,0)	72.40	77.40	83.30
(1,0,0,0.3)	43.83	47.78	59.60
Alignment quality. Finally, to measure the alignment quality of paired images, we compare the Percentage of Correct Keypoints (PCK) using LoFTR [42] as a keypoint matcher by following the methodology presented by Feng et al. [13]. A keypoint pair is considered correctly aligned when  $d < \alpha \times \max(H, W)$ , where  $d$  is the position difference between a pair of matched keypoints,  $\alpha$  is the
Table 4: Restoration performance for synthetic and real UDC datasets. The term Input refers to the PSNR and SSIM values between the paired degraded and ground-truth images.
Input	DISCNet [15]	UDC-UNet [31]	Uformer-T [47]	ECFNet [54]	SRGAN [29]
Feng-S [15]	PSNR	26.08	43.27	49.37	42.47	52.17	32.35
SSIM	0.8561	0.9877	0.9933	0.9844	0.9958	0.9538
UDC-SIT	PSNR	21.03	26.32	27.44	27.28	28.22	24.70
SSIM	0.7330	0.8457	0.8637	0.8594	0.9002	0.8195
threshold, and  $H$  and  $W$  represent the height and width of the image. To apply a consistent alignment criterion across the datasets with different resolutions, we uniformly set  $\max(H, W) = 1024$ , aligning with Feng et al. [13]. Then the PCK for an image pair is defined by the ratio of the number of correctly aligned keypoint pairs to the total number of keypoint pairs. Table 2 illustrates the comparison result. Since Zhou-S and Feng-S consist of synthetic images, they do not require an additional alignment process, resulting in PCK values close to  $100\%$ . On the other hand, Feng-R undergoes alignment using AlignFormer on the images in the paired dataset. It achieves  $58.75\%$  for  $\alpha = 0.01$ . Unlike Feng-R, UDC-SIT consistently exhibits PCK values close to  $100\%$  across all values of  $\alpha$ .
Table 3 shows the analysis result of PCK values on average for various  $\lambda_{1}$ ,  $\lambda_{2}$ ,  $\lambda_{3}$ , and  $\theta_{\mathrm{rotation}}$  combinations. We found that the optimal combination of  $\lambda_{1}$ ,  $\lambda_{2}$ ,  $\lambda_{3}$ , and  $\theta_{\mathrm{rotation}}$  is different for each image pair. The images in UDC-SIT are selected by human inspection, and the PCK values of UDC-SIT in Table 2 originate from the human assessment. The human inspection result is comparable to results obtained by the best combination (1, 0, 1, 0) in Table 3.
Applying rotation (i.e., the case of  $(1,0,0,0.3))$  from Algorithm 1 significantly reduces PCK as shown in Table 3. While this rotation causes a slight MSE decrease, its impact on digitized images leads to notable PCK decline because of changes in ordinary translational correlation during the rotation [7].
To mitigate rotation and tilt effects, we use an image-capturing system in Figure 2, applying only shifts in  $(\mathrm{x},\mathrm{y})$  coordinates. Employing the DFT loss without MSE (the cases of  $(0,1,0,0)$  and  $(0,0,1,0)$ ) yields lower PCK values of  $34.05\%$  and  $62.27\%$ , respectively ( $\alpha = 0.01$ ). Using only MSE and  $\Delta \mathcal{F}_{amp}(u,v)$  without  $\Delta \phi(u,v)$  (case  $(1,1,0,0)$ ) results in a reduced PCK of  $35.55\%$  ( $\alpha = 0.01$ ). The loss functions encompassing spatial and frequency domains, focusing on  $\phi(u,v)$  of  $\mathcal{F}(u,v)$  (the cases of  $(1,0,1,0)$  and  $(1,1,1,0)$ ), show high PCK values of  $98.22\%$  and  $86.13\%$ , respectively ( $\alpha = 0.01$ ). The case of  $(1,0.1,0.1,0)$  lacks sufficient  $\lambda_{2}$  and  $\lambda_{3}$  to reflect the frequency domain loss effectively.
5 Effects on learnable restoration models
We demonstrate the effectiveness and benefits of UDC-SIT by comparing the UDC image restoration performance with Feng-S. We only compare UDC-SIT to Feng-S because Yoo-S is not publicly available, Feng-R is planned for a future release but is currently unavailable, and Zhou-S needs more flares that are essential for UDC image restoration. We use five learning-based image restoration models, including DISCNet [15], UDC-UNet [31], Uformer [47], ECFNet [54], SRGAN [29] using UDC-SIT. We modify the authors' code of the models to account for the difference in dynamic range and channel size between UDC-SIT and Feng-S [15]. To ensure reproducibility, we explain the experimental setting in the appendix and our GitHub.
The restoration performance of the five models on Feng-S and UDC-SIT is given in Table 4. Models trained with Feng-S successfully restore the images with up to 2.0 times higher PSNR values than Input (except SRGAN). They also achieve a nearly perfect SSIM score. However, none of the models trained with UDC-SIT can restore the images to the same extent as Feng-S. Feng-S inadequately represents actual UDC degradation, such as intense flares. The high degree of glares, shimmers, and streaks results in substantial information loss. Consequently, model performance degradation occurs when those obscured objects become unrecognizable. Thus, restoration becomes more challenging with UDC-SIT than with conventional synthetic datasets.
Effects of annotations. It is noteworthy that annotations are specifically available in the UDC-SIT. Flare scenes heavily rely on the capturing environment. For example, artificial lights emit a different
Figure 8: Description of flare components. (a)-(c) The components of a flare include glare, shimmer, and streaks [11]. (d)-(f) Flare components observed in actual degraded images of UDC-SIT. (g)-(i) Ground-truth images corresponding to the UDC degraded images.
spectrum than natural sunlight, resulting in a distinct diffraction pattern. Weak light sources can cause streaks under low-light conditions. Thus, we attach annotations, such as light source, day/night, indoor/outdoor, and flare components, to the images in UDC-SIT. UDC-SIT follows the classification labels of flare components by Dai et al. [11] including shimmer, streak, glare, and light source as illustrated in Figure 8. Detailed explanations and instructions for the annotations, and experimental result of their effect on the restoration models can be found in the appendix.
The degradation of images by the UDC depends on the display pixel patterns, which vary between products, as shown in Figure 6. When obtaining UDC-SIT, Galaxy Z Fold 3 [39] display panel is used. Thus UDC-SIT is optimal for restoring Galaxy Z Fold 3 [39] images. Models trained with UDC-SIT may not be suitable for restoring UDC images taken by other devices (e.g., ZTE Axon 20 [10] and Galaxy Z Fold 4 [40]).
As far as we know, UDC-SIT is the first dataset to include real-world UDC degradation, such as low transmittance, blur, noise, and flare, along with detailed annotations for the light source, day/night, indoor/outdoor, and flare components. With UDC-SIT, one can train a UDC image restoration model to improve the quality of UDC images taken by the UDC. We propose an effective image-capturing system for paired UDC-distorted and ground-truth images. We also propose a technique for aligning the paired images by exploiting the discrete Fourier transform. The experimental result of comparing UDC-SIT and a representative synthetic UDC dataset with five representative learnable restoration models indicates that the models trained with the synthetic UDC dataset are impractical because the synthetic UDC dataset does not reflect the actual characteristics of UDC-degraded images. This implies UDC-SIT is an adequate dataset for UDC image restoration.
This work was supported in part by the Institute for Information & communications Technology Promotion (IITP) grant (No. 2018-0-00581, CUDA Programming Environment for FPGA Clusters) and by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. RS-2023-00222663). This work was also supported in part by the Samsung Display Co., Ltd. ICT at Seoul National University provided research facilities for this study.
[45] Zachary Teed and Jia Deng. “Raft: Recurrent all-pairs field transforms for optical flow”. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer. 2020, pp. 402–419.
[46] Shitong Wang et al. "Distortion correction in surface defects evaluating system of large fine optics". In: Optics Communications 312 (2014), pp. 110-116.
[47] Zhendong Wang et al. "Uformer: A general u-shaped transformer for image restoration". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 17683-17693.
[48] Yicheng Wu et al. "How to train neural networks for flare removal". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 2239-2247.
[49] Qirui Yang et al. "Residual and dense unet for under-display camera restoration". In: Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part V. Springer. 2021, pp. 398-408.
[50] Jewon Yoo et al. “7-5: Synthetic Dataset for Improving UDC Video Restoration Network Performance”. In: SID Symposium Digest of Technical Papers. Wiley Online Library. 2022, pp. 58–60.
[51] Syed Waqas Zamir et al. "Restormer: Efficient transformer for high-resolution image restoration". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 5728-5739.
[52] Yuqian Zhou et al. "Image restoration for under-display camera". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 9179-9188.
[53] Yuqian Zhou et al. "UDC 2020 challenge on image restoration of under-display camera: Methods and results". In: Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16. Springer. 2020, pp. 337-351.
[54] Yurui Zhu et al. "Enhanced coarse-to-fine network for image restoration from under-display cameras". In: Computer Vision-ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. Springer. 2023, pp. 130-146.
In this section, we discuss the factors that contribute to the UDC degradation. We also explore the concept of dynamic range and tone mapping. Also, we discuss color representation and image formats to elucidate the rationale behind collecting the UDC-SIT dataset in the form of RAW images with an GRBG 4-channel Bayer pattern.
A.1 Image degradation due to the UDC
The representative reasons for image degradation due to the UDC can be categorized into three classes: noise, low transmittance, and flare caused by the display panel on top of the camera lens. The reasons for the degradation differ depending on various factors, such as display types (T-OLED and P-ILED), pixel designs, the distance from the display to the camera lens, and the lens structure. Qin et al. [35] demonstrates that pixel designs affect the diffraction level. The optimal pixel design could improve the image quality of the UDC. However, the design of the pixels needs to consider many factors, not only the degradation by the UDC but also the size and shape of the red, green, and blue sub-pixels. They affect the color intensity and image sharpness.
Under the UDC setting, because the pixels work as if they were slits, as shown in Figure A.1, the light diffracts, and this causes the degradation. The light diffracts when it passes through obstacles comparable in size to its wavelength. Unfortunately, we cannot avoid diffraction because the aperture size of the pixel layout is on the order of the wavelength of visible light. The diffracted light propagates to the camera sensor through the camera lens.
The Point Spread Function (PSF) of an optical system is the image-irradiance distribution or impulse response on an image plane that results from a point source [5]. For example, an image of a distant star through a telescope is a PSF. It is an energy distribution pattern. It peaks in the center and has characteristics of long-tail low-energy sidelobes. Since a different light source has a different PSF, the flare shape might differ depending on the light source. Also, the diffraction levels of red, green, and blue waves differ because of their wavelengths in the visible ray region.
A.2 Dynamic ranges and tone mapping
An image's dynamic range describes a scene's luminance range or the limits of the luminance range that a digital camera can capture [33]. The digital camera sensor is narrower than the range of brightness that the human eye can accommodate. Multiple frames of the same scene captured with different exposures, shutter speeds, and F-numbers can be combined to create an image with a higher dynamic range than individually captured frames.
Tone mapping is required to view a video or image captured with High Dynamic Range (HDR) on a monitor. Since the display device has a limited dynamic range unsuitable for reproducing the full range of light intensities in an actual scene, tone mapping approximates the appearance of an HDR image in a medium with a more limited dynamic range by mapping one set of colors to another. One of the representative tone mapping operations is Reinhard tone mapping [36].
(b)
Figure A.1: (a) The diffraction. (b) The PSF is attributed to the pixel pattern on the UDC area.
A.3 Color representation and image formats
A Bayer pattern [4] is a Color Filter Array (CFA) for arranging RGB color filters on a square grid of photosensors. It uses a checkerboard arrangement of filter patterns, such as BGGR, RGBG, GRBG, or RGBB. Image sensors for digital cameras typically use the Bayer pattern. Since each pixel provides only a single-color pixel value (e.g., green) obtained from a CFA sensor, demosaicing is used to reconstruct the full-color RGB image. Interpolation computes the pixel's missing color values (e.g., red and blue) using the colors of neighboring pixels.
RAW image files contain unprocessed or minimally processed data from a digital camera's image sensor. RAW image files in recent smartphone camera sensors have a 10-bit range, and each pixel records one of the three colors: red, green, and blue in 1,024 steps using the Bayer pattern. Other image formats, such as JPEG and PNG, are converted from the RAW format using an Image Signal Processor (ISP) with some loss of information in 256 steps. Thus, collecting the images in the RAW format is essential to ensure that the dataset is free from the influence of the ISP (i.e., free from the loss of information).
B Details of the UDC datasets and DNN models used
This section provides detailed information on the UDC datasets and the DNN models used to compare the datasets in the paper for reproducibility.
The datasheets for datasets describe UDC-SIT's hosting, licensing, and maintenance plan. The UDC-SIT dataset, code, and evaluation procedure can be found and downloaded at: https://github.com/mcrl/UDC-SIT.
B.1 UDC datasets
The images in UDC-SIT are captured in the RAW format using the GRBG 4-channel Bayer pattern, effectively eliminating the influence of the built-in ISP. Although HDR captures the details in shadows and highlights better, actual UDC images are predominantly in LDR. Consequently, we capture images in LDR to align with real-world conditions.
We compare the available UDC dataset in Table B.1. Unlike Table 1, it emphasizes information relevant to the DNN model execution. Zhou et al. [52] also obtain 16-bit RAW sensor data but only release the paired RGB data. Thus, we classify Zhou-S as 3-channel RGB images. When working with HDR images, tone mapping becomes necessary. Feng et al. capture 330 images with dimensions  $3200 \times 2400 \times 3$ . These are then cropped into 6,521 training images of size  $512 \times 512 \times 3$  and 226 test images sized  $1024 \times 1024 \times 3$ . We provide the dataset in the NPY format (.npy) to make UDC-SIT open and widely used.
B.2 DNN models for evaluating UDC-SIT
The DNN models used for evaluating UDC-SIT include DISCNet [15], UDC-UNet [31], Uformer-T [47], ECFNet [54], and SRGAN [29]. While we primarily adhere to the original authors' code for the models, we make some modifications.
Table B.1: Comparison of the available UDC datasets.
Zhou-S [52]	Feng-S [15]	Feng-R [13]	UDC-SIT
Raw sensor data	✓	✓
Color representation	RGB	RGB	RGB	GRBG Bayer
Dimensions	[1024, 2048, 3]	[800, 800, 3]	[512, 512, 3], [1024, 1024, 3]	[1792, 1280, 4]
Dynamic range	LDR	HDR	HDR	LDR
Tone mapping	✓	✓
Value range	[0, 255]	[0, 500]	[0, 255]	[0, 1023]
File format	.png	.npy	.png	.npy
DISCNet, UDC-UNet, and ECFNet are designed to restore UDC images in HDR (i.e., Feng-S [15]), while Uformer-T is designed to restore motion-blurred images in LDR. Also, SRGAN is designed to super-resolve at large upscaling factors while also taking into account the preservation of high-frequency details. Consequently, we make some necessary adjustments to the PyTorch DataLoader, implementing tone mapping [36] for Feng-S and opting for normalization instead of tone mapping for UDC-SIT. Also, we increase the channel size of the models to 4 to match the channel size of UDC-SIT. Training is performed with four NVIDIA GeForce RTX 3090 GPUs. The following are the training details specific to each model for the UDC-SIT dataset:
- DISCNet. Due to the channel size of 4 in UDC-SIT, we solely utilize L1 loss as the loss function, excluding perceptual loss. To accelerate training, the models are trained using PyTorch Distributed Data Parallel (DDP) [30] with a batch size of 32.
- UDC-Unet. The model output is clamped to the range [0, 1] to calculate the training loss. The additional modification aligns with the details described in DISCNet.
- Uformer-T. To maintain consistency with other models operating solely in FP32, we have modified Uformer-T to use FP32 instead of mixed precision. Also, we utilize Transformer feature channels with  $C = 8$ . The patch size is set to  $1, 280 \times 1, 280$ , and the batch size is 4.
- ECFNet. Since the training script of ECFNet is not publicly available, we follow the method used in MIMOUNet [9] that heavily affected ECFNet. We increase the base channel size of blocks from 24 to 32 to provide eight channels per image channel. ECFNet introduces the progressive training strategy adopted by Restormer [51]. Specifically, the training phase of the network consists of three stages with different patch sizes (e.g.,  $256 \times 256$ ,  $512 \times 512$ , and  $800 \times 800$ ). We modify the batch size of each stage from 3, 1, and 2 to 16, 4, and 4, respectively. In the third stage, we employ activation recomputation [8] to address the memory constraint of NVIDIA GeForce RTX 3090 GPUs. It effectively reduces the peak memory usage from 46GB to 18GB per device, enabling efficient utilization of available GPU memory resources. We do not use external data or adopt model ensemble strategies that the authors of the models describe.
- SRGAN. The model output is clamped to the range [0, 1] to calculate the training loss. Due to the channel size of 4 in UDC-SIT, we utilize L2 loss and adversarial loss as the loss function, excluding perceptual loss.
C Details of the annotations
This section describes annotation instructions for crowdsourcing and the effect of annotations in the UDC-SIT.
Table B.2: Annotation examples for UDC-SIT. The corresponding annotated images are shown in Figure B.2. "Indoor" and "Outdoor" scenes are labeled 1 and 3, respectively. The labels for "No distinction," "Day," and "Night" are 1, 2, and 3, respectively. For indoor images, if flares occur due to the influence of natural light sources through windows, we label them "Day" and "Night". For indoor scenes without windows, we label them "No distinction". Glares, shimmers, and streaks are labeled 1 if present and 0 otherwise. If there is no flare, it is labeled 0. The occurrences of flares caused by natural sunlight, artificial light, or both are labeled 1, 2, and 3, respectively.
File	Indoor/Outdoor (1/3)	No distinction/Day/Night (1/2/3)	Glare (0/1)	Shimmer (0/1)	Streak (0/1)	Light sources (0/1/2/3)
317.npy	3	3	1	1	0	2
333.npy	3	3	1	1	1	2
677.npy	3	2	1	0	0	1
689.npy	3	2	1	1	1	1
698.npy	1	2	1	1	0	1
903.npy	1	2	1	1	0	2
905.npy	1	1	1	1	0	2
2384.npy	3	2	0	0	0	0
(a)
(b)
Figure B.1: Visual comparison of flare removal on real-world UDC images. Flare shapes vary according to the captured environments. GT stands for ground truth. (a) Indoor + artificial light (glare + shimmer). (b) Indoor + artificial light (glare + shimmer + streak). (c) Outdoor + day + sunlight (glare + shimmer + streak). (d) Outdoor + night + artificial light (glare + shimmer + streak).
Table B.3: Annotation distribution and the number of pairs.
Label	# of pairs
Indoor (1)	1,754
Outdoor (3)	586
No distinction (1)	1,340
Day (2)	649
Night (3)	351
Glare (0 or 1)	2,037
Shimmer (0 or 1)	1,899
Streak (0 or 1)	1,067
No flare (0)	273
Natural light (1)	175
Artificial light (2)	1,639
Both (3)	253
Total	2,340
C.1 Effects of image capturing environments
The combination of various factors, including light sources, indoor/outdoor, and day/night, affect the level of degradation or flare shapes in UDC images. The images in Figure B.1(a) show only glare and shimmer, while the images in Figure B.1(b) display glare, shimmer, and streak caused by different artificial light sources. Natural sunlight in Figure B.1(c) has the strongest intensity. It shows extreme flares with glare, shimmer, and streak even in the daytime, while artificial light at night with the low-light condition in Figure B.1(d) has relatively weak intensity but still has extreme glare, shimmer, and streak.
C.2 Types of annotations
We offer annotations for each image pair. Table B.3 provides a detailed overview of the total count and distribution of different annotation labels. Note that an image pair can have multiple annotation labels. The parenthesized number beside a label is the encoding of the label. The pairs are categorized based on image degradation factors, such as indoor/outdoor, day/night, glare/shimmer/streak, and light sources. If images are captured indoors without any window or access to natural sunlight, there would be no distinction between daytime and nighttime. In such cases, we label them as "No distinction." Detailed information can be found at: https://github.com/mcrl/UDC-SIT.
Participants involved in the crowdsourcing for UDC-SIT are provided with the annotation examples in Table B.2. The corresponding annotated images are shown in Figure B.2. The details of the annotations are described as follows.
Flare components. We follow the classification scheme proposed by Dai et al. [11] to categorize different flare components, including glares, shimmers, and streaks. Figure 8 in the main body of the paper and Figure B.2 provide detailed explanations and visual representations of each flare component, serving as comprehensive annotation instructions. A glare refers to the presence of intense and strong light that produces artifacts such as circular light patterns. A shimmer involves rapid and subtle light or color intensity variations within an image. A streak is a long, thin, and
GT
Degraded
GT
Degraded
Figure B.2: UDC-SIT in various scenarios. Table B.2 shows the corresponding annotations. (a) 317.npy. (b) 333.npy. (c) 677.npy. (d) 689.npy. (e) 698.npy. (f) 903.npy. (g) 905.npy. (h) 2384.npy.