UDC-SIT: A Real-World Dataset for Under-Display Cameras
Authors: Kyusu Ahn
Byeonghyun Ko
HyunGyu Lee
Chanwoo Park
Jaejin Lee
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Under-display camera
Image restoration
Real-world dataset
Alignment of paired images
Fourier transform

Abstract: Under Display Camera (UDC) is a novel imaging system that mounts a digital camera lens beneath a display panel with the panel covering the camera. However, the display panel causes severe degradation to captured images, such as low transmittance, blur, noise, and flare. The restoration of UDC-degraded images is challenging because of the unique luminance and diverse patterns of flares. Existing UDC dataset studies focus on unrealistic or synthetic UDC degradation rather than real-world UDC images. In this paper, we propose a real-world UDC dataset called UDC-SIT. To obtain the non-degraded and UDC-degraded images for the same scene, we propose an image-capturing system and an image alignment technique that exploits discrete Fourier transform (DFT) to align a pair of captured images. UDC-SIT also includes comprehensive annotations missing from other UDC datasets, such as light source, day/night, indoor/outdoor, and flare components (e.g.,  shimmers, streaks, and glares). We compare UDC-SIT with four existing representative UDC datasets and present the problems with existing UDC datasets. To show UDC-SIT's effectiveness, we compare UDC-SIT and a representative synthetic UDC dataset using four representative learnable image restoration models. The result indicates that the models trained with the synthetic UDC dataset are impractical because the synthetic UDC dataset does not reflect the actual characteristics of UDC-degraded images. UDC-SIT can enable further exploration in the UDC image restoration area and provide better insights into the problem. UDC-SIT is available at: https://github.com/mcrl/UDC-SIT.

UDC-SIT: A Real-World Dataset for Under-Display Cameras
Kyusu Ahn
Byeonghyun Ko² HyunGyu Lee² Chanwoo Park2
Jaejin Lee
'Dept. of Data Science, Seoul National University, Seoul, Republic of Korea
"Dept. Computer Science and Engineering, Seoul National University, Scoul, Republic of Korea
"Research Center, Samsung Display Co., Ltd., Yongin, Republic of Korea
amu.aa.kks (byeonghyun, hyungyu) @aces.snu.ac.kr {99chanwoo, jaejin) @smu.ac.kr
Abstract
Under Display Camera (UDC) is novel imaging system that mounts digital camera lens beneath display panel with the panel covering the camera. How- ever, the display panel causes severe degradation to captured images, such as low transmittance, blur, noise, and flare. The restoration of UDC-degraded images is challenging because of the unique luminance and diverse patterns of flares. Existing UDC dataset studies focus on unrealistic or synthetic UDC degradation rather than real-world UDC images. In this paper, propose real-world UDC dataset called UDC-SIT. To obtain the non-degraded and UDC-degraded images for the same scene, propose an image-capturing system and an image align- ment technique that exploits discrete Fourier transform (DFT) to align pair of captured images. UDC-SIT also includes comprehensive annotations missing from other UDC datasets, such as light source, day/night, indoor/outdoor, and flare components (e.g., shimmers, streaks, and glares). We compare UDC-SIT with four existing representative UDC datasets and present the problems with existing UDC datasets. To show UDC-SIT's effectiveness, we compare UDC-SIT and representative synthetic UDC dataset using five representative learnable image restoration models. The result indicates that the models trained with the synthetic UDC dataset are impractical because the synthetic UDC dataset does not reflect the actual characteristics of UDC-degraded images. UDC-SIT can enable further exploration in the UDC image restoration area and provide better insights into the problem. UDC-SIT is available at: https://gthubb.om//er//DDDDCC
Introduction
Under Display Camera (UDC) is technology designed to place camera module under the display to use the UDC area as a display space and take pictures when the camera operates. Since a larger screen-to-bel to-body ratio is common consumer demand that leads to trend toward bezel-less display products (2T. 26 [T]. such as smartphones, laptops, tablets, and TVs, the display products equipped with UDC will well meet such market trend. Moreover, video conferencing uses UDC products that enable natural eye-tracking by arranging a camera in the center of the display. However, UDC has some drawbacks of image deterioration problems, such low transmittance, blur, noise, and flare.
Since the pixels the micrometer scale diffract the light traveling through the camera lens (35), degrading image quality, UDC displays have a lower pixel density above the camera lens to reduce transmittance loss and diffraction, as shown in Figure [ However, lower display resolution the UDC area affects natural video viewing. Thus, improving the quality of UDC images is critical problem. Especially UDC image reconstruction models to enable higher pixel densities around the camera essential to overcome the problem for more comfortable viewing.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

(a)
(b)
Encap
Camera
EML
LTPS
hole
Substrate
Normal; UDC
Normaa
Camera
Figure 1:
Comparison of conventional hole-display and under-display cameras. The UDC area has lower pixel density since the pixel pattern functions as diffraction slits. (a) Under Display Camera (UDC). (b) Hole display camera. (c) Pixel structure of the UDC area. (d) Comparison between the UDC area and other display area.
Several studies address the UDC image restoration problem. However, most of them 13 50
52
have limitations because their datasets do not completely reflect the properties of actual UDC images (i.e., are synthesized). challenging problem in constructing the UDC dataset finding a matching pair of the ground truth and distorted UDC images, which requires significant time and effort.
Feng el al. (13) proposes seeudd-real-wordee UDC dataset. However, their dataset contains images with occlusions. Ignatov al. [23 24 try match two images for the same scene captured by different devices with different positions and angles. They use the SIFT key points (32) of paired images and the RANSAC algorithm [16] to align the two images in pair. However, the geometric alignment algorithm fails to adequately perform when there a large discrepancy the scenes the two images, such UDC degradation.
(a) camera module
(b)
Samsung Galaxy Note 10
UDC area
The lid can
be openedd
The lid
or elosed
The lid
is opened
The display puncFod
Samsung Galaxy Z- Fold
The image capturing system to capture pair of matching images (the ground-truth image and the degraded image by the UDC). (a) The lid opens to get ground-truth image. (b) The lid closes to get the corresponding degraded image.
Figure 2:
This paper proposes new UDC dataset called UDC-SIT (UDC's Still Images by the Thunder Research Group), As far as we know, the first real-world UDC dataset to overcome the problems of the existing UDC datasets.
We devise an image-capturing system that minimizes lens movement when capturing matching image pairs. We cut and take the UDC area of the flexible display panel of a smartphone with UDC (c.g.. Samsung Galaxy Z-Fold (39]) We the piece of the display panel
lid on the camera lens of (38 The lid on the non-UDC
another smartphone with non- UDC (e.g., Samsung Galaxy Note 10
can be opened and closed. As a result, our capturin system has minor geometric inconsistency compared to Feng et al. [13). In addition, it helps to restrict rotations and tilts by the movement of lid a degree that does not notably influence alignment. Figure Figure@illlssraate illustrates our image-capturing system. We get matching pair by capturing the ground-truth image with the lid off and the distorted image with lid on.
We compensate for the exploiting the discrete Fourier transform (DFT) (6). The discrepancy in pixel positions is inevitable because of the slight movement of the camera when opening and closing the lid.
difference between the two images for the same scene by
The contributions of this paper summarized follows:
We propose an image-capturing system for obtaining matched pairs of undistorted and UDC-distorted images for the same scene.
We propose an image aligning technique for the paired images of the same scene by exploiting the discrete Fourier transform.

---

We provide real-world UDC dataset, UDC-SIT, which accurately reflects actual im- age degradations by the UDC. UDC-SIT offers rich set of annotations that stim- ulate research in UDC image restoration. We make UDC-SIT publicly available hhtps://gihuuu.ccmmmmm/l/CCIICC
To show UDC-SIT's effectiveness, compare UDC-SIT and representative synthetic UDC dataset using five representative learnable image restoration models.
2
Related work
Image restoration. Image restoration techniques restore a high-quality image from a degraded one. They include denoising, deblurring, and flare removal tasks. Camera-captured images often contain noisy and blurred pixels due focus errors and incorrect light sensitivity. Restoring these images by removing noise and blur is a common task. Uformer [47 is one of the leading hierarchical transformer- based models for restoring such images. SIDD [T] comprises 30,000 noisy images captured by five smartphone cameras in 10 different scenes under various lighting conditions. DND (34) offers 50 pairs of noisy and noiseless images of very high resolution.
Light flares degrade image quality significantly due to strong light intensities. Wu el al. 488 generate synthetic flare removal datasets and train a neural network by modeling the optical characteristics of flares. Also, Dai et al. 12) introduce a nighttime flare removal dataset to address the limitations of existing methods that only work well on daytime flares.
The UDC image restoration is complex, and its degradation patterns differ from other restoration tasks of images captured by standard cameras.
Existing UDC datasets. Zhou et al. [52 tackle the UDC image restoration problem using paired images from Monitor Camera Imaging System (MCIS) and synthesized PSFs using optical modeling. However, their dataset has limitations, including unrealistic flares captured from a monitor with a limited dynamic range and inaccurate PSFs. They provide only 300 pairs of images for T-OLED and P-OLED, respectively.
Feng et al. [5] improve the UDC dataset. To measure the PSF, they place white point light source one meter away from the OLED display of ZTE Axon 20 (10] and following the methodology presented by Sun el al. 43
By convolving the PSF with HDR images from the HDRI Haven dataset [18), a synthetic dataset is generated. However, it lacks real-world characteristics because the images and PSFs come from different devices. Also, the flare shapes are limited, not including the distorted flares addressed by Yoo et al. [50].
Yoo et al. (50] introduce a synthetic UDC dataset that includes spatially varying PSFs obtained by optical simulation using the Brown-Conrady Distortion model [46]. However, their simulated flare distortions differ from real-world distortions, and their dataset is not publicly available.
Feng al. [13 create pseudo-real dataset by capturing degraded images with ZTE Axon 20 UDC [10] and ground-truth images with iPhone 13 Pro camera (25) They face domain discrepancy and geometric misalignment challenges. Geometric misalignment is severe due to using different camera modules for the two paired images. AlignFormer [3] overcomes the geometric misalignment in UDC by aligning domain information with StyleConv (28) and AdalN (22), and geometric information with an attention mechanism and pre-trained optical flow estimator called RAFT 45)
DNN models for UDC image restoration. The UDC image restoration challenges 14 53 use the datasets by Feng et al. [15] and Zhou et al. [52]. In these challenges, most of the top-ranked teams 31
44
49 54] use the U-Net model (37] and residual networks [19] as their backbone Deep Neural
Network (DNN) model for restoration.
3
Obtaining aligned images
This section describes our alignment technique of the standard and UDC images for the same scene. Misalignment between the two images is not due to the UDC itself but rather a problem that arises when capturing them. Previous techniques, such as SIFT (32 and RANSAC (16), are inadequate for this purpose because of severe degradation by the UDC. Feng et al. B3 use two cameras with
3

---

different specifications. leading to variations in perspectives and contents between paired images. Some tools, such AlignFormer (13), align these images but introduce occlusion regions.
In contrast, our approach minimizes misalignment without introducing occlusion regions. After capturing the two images with image-capturing system, exploit discrete Fourier transform (DFT) to align the two paired images. We especially exploit their spatial frequency domain after the DFT achieve Ingrradttion-rsillee alignment 9 17 27
3.1 Discrete Fourier transform
DFT converts a discrete signal represented by complex exponential waves into constituent frequencies. Equation I defines 2D DFT used to obtain the frequency representation of an image.
M-1N-1 F(u, - ΣE -
(( 12x
(1)
V=0
where F(u, v) denotes the frequency value, (u, v) represents point in the frequency domain, M x N the image size, and f(z, y) is the pixel value at point (z, in the image (i.e., the spatial domain). Using Euler's 's formula decomposes the exponential function in Equation intto cosine and sine functions, and Equation becomes
F(u, v) ((,,,)
v) + (I(,,
where R(u, v)) and I(u, u) denote the real and the imaginary part of F(u, v), respectively. The amplitude [F v)] and phase o(u,v) v) of F(u, v) are defined as:
I(u, v) R(u, v)
[F(u, w) = [R² (u, v) I22u,, wi]
and olu, v) = tan
(3)
3.2 Alignment of paired images
Let M N be the image size. To measure the difference between the degraded image (D) and ground-truth image (G), we typically use Mean Squared Error (MSE).
M-1N-1
MSE = Σ Σ (D(z,y) -(zz))))
(4)
S=0 ==0
However, MSE primarily emphasizes local information points (ær, 1/). In contrast, the spectrum's value for specific point (u, v) in the frequency domain relies on the collective contribution of all points (x, y) in the spatial domain because F(u, v) is the sum that iterates through each pixel (æ; y) of the image in Equation [ Thus, to align the two images, we assess the distance between paired images in both spatial and frequency domains shown in Figure 3 To incorporate both local and global information, employ a loss function that combines information in the spatial and frequency domains:
M-1N 1-1
M-1 1-1
M-IN- Σ Ao(u,)) (5)
Loss A
Σ 200 m-0
E((e, v) -(x,,))
Σ
Σ UnD
AF (u, Au
where AFamp(u v) is the L1 distance for the amplitude defined as AF AFomp(u, v) FFD v) Fo(u, v)}, and Ad(u,v) is the Il1 distance for the phase defined Ap(u,)) jp (o(u, v)).
Flu, v) at point (u, 2) represents distinct spatial frequency component, and applying inverse DFT to Flu, v) at a point (u, v) generates a sinusoidal grating in the spatial domain. Figure 4aa and Figure (c) illustrate the isolation of sinusoidal gratings associated with Fo v) and Fp v). respectively.
The significant advantage of using DFT in aligning the original and shifted images lies in its capability
to decompose an image into its constituent spatial frequency components. Figure and Figure ec)
display the initial three low-frequency sinusoidal gratings from the two images, clearly revealing distinct spatial shift. Notably, while the amplitude of sinusoidal gratings remains constant within the same column of Figure a) and Figure (c), their corresponding phases exhibit variations. Figure(Db

---

Spatial domain
Frequency domain
DFT
DFT
Imput
FG
FD
ØG
Φ
y
X
U
MSE
AJ amp
AΦ
Figure 3: Overview of the proposed loss function to align two paired images. The loss function considers the difference between degraded (D) and ground-truth (G) images in both the spatial and frequency domains.
(b)
y
x
(d)
shifted
+Y +y
X
11
Figure 4: Frequency analysis of two paired images. One is the original image (G), and the other is the spatially shifted image (D) of G. Let Fx (u, v) be the result of applying DFT to image X. (a) G in the spatial domain comprises multiple sinusoidal gratings. Each sinusoidal grating results from applying inverse DFT Fa(u, v). (b) The amplitude difference between G and D. There is no difference. (c) Similar to G, D in the spatial domain comprises multiple sinusoidal gratings. (d)
The phase difference between G and D.
and Figure 4 visually depict the differences in amplitude (Figure 4b)) and phase (Figure 4d) between the two images. Reducing AΦ is crucial for effectively aligning a misaligned pair of images for the same scene. Figure is conceptual illustration for paired images involving shifts without degradation. However, due to the degradation, UDC-degraded images also exhibit differences from the ground truth in amplitude. Figure elucidates the importance of the phase component in the alignment the images.
Our alignment algorithm minimizes the loss in Equation between the ground-truth image and the degraded image. The misaligned degraded image rotated, shifted and then cropped to achieve the same size as the ground truth until the loss reaches the minimum. Handling rotation and shifts are manageable, but addressing tilt presents challenge. It because tilt requíres perspective transforms optimized for objects within single image sharing the same plane. Thus, our current emphasis is on tackling shifts and rotations, with tilt considerations excluded. Nonetheless, our data collection is meticulous, limiting rotations and tilts to degree that does not notably influence the alignment, as the PCK values Table afffirmm
The original size of camera- captured images is (2016, 1512, 4). The ground truth image is center- cropped (1792, 1280, 4). The degraded image similarly cropped around the center. For the degraded image, iterative shifting of (x, y) coordinates and rotation are used to find the minimum loss point where the cropped degraded image aligns with the cropped ground truth image. The final cropped image size becomes (1792, 1280, 4) ensure H and W are multiples of 256. The detailed algorithm illustrated in Algorithm [ Here, we establish s 20, 0.3, r 0.1, and (Ai, , d2 Aa) combinations in Table as hyperparameters.
5

---

Algorithm Alignment of paired images Ig and ID
Require: Images Ic. Ln of size (H, W). hyperparameters Or, Yn, X2, a
Ensure: Aligned images Ca. CD of size (H", W*)
Crop CG from 1G using center crop
Crop Cp from To to the size of Cg
Initialize best loss Thest to large value
Initialize optimal shifts Sapt Sopt y, and rotation Ooye to for Ovesioo from 0, to 0, with step
Apply rotation of Oranrion to Ip to get Drouss for Tean from -s to with step do for Yshi from to with step do Calculate crop position (p, 9) relative the center crop:
p = Zoenier
T-hin
9. Yeemet srop + Main
Crop image Cuunp from I Daaaes at position (p. 9)
Calculate loss I. using the loss function in Eq. between CDup and Ca
if Luese then
Update Luest to
Update Soptx to 4shin Update Sopty VAii Update fogg tronssion
end
end for end for
end for
Apply optimal rotation Puge In get Draued
Calculate crop position (Pops: (ops) relative to the center crop:
Pupi Zventer stup Soptxx
Topt Vcenter_ccpp + Supt
Crop P acquire aligned image Cp position (Popty 2opt)
We capture images in UDC-SIT without any Image Signal Processing (ISP) and in RAW format. While High Dynamic Range (HDR) captures more details in shadows and highlights, authentic UDC images typically in Low Dynamic Range (LDR). Generating HDR images requires capturing multiple LDR images with different exposures, then combining them to create an HDR image (2). This process differs from general photography, so we gather our dataset LDR.
Table I: Comparison of the UDC datasets. Unlike others, UDC-SIT provides annotations, such as light source, day/night, indoor/outdoor, and flare types. Flare types are classified as shimmer, streak, and glare.
Datasct
Scene
Dynamic range
Dataset size
Annotations
Publicly available
Zhou et al.
52
Synthetic
1.DR
300
No
Yes
Feng et al.
15
Synthetic
HDR
2,376
No
Yes
Yoo et al.
50
Synthetic
LDR
No
No
Feng et al.
13
Real
HDR
6.747
No
Yes
UDC-SIT
Real
L.DR
2,340
Yes
Yes
4 Comparison with the existing UDC datasets
In this section, we compare UDC-SIT with the four existing representative UDC image datasets. We summarize the five datasets Zhou-S (52], Feng-S [15). Yoo-S [50), Feng-R [13, and UDC-SIT in T where S and R stand for synthetic and real datasets, respectively. Feng et al. capture 330 images and then crop them into 6,7 7,77 small patches. We explain the detail in the appendix.
Noises and transmittance decrease. Under low-light conditions, the camera sensor amplifies both the desired signal and unintended random noise. Since the camera sensor the UDC positioned beneath the display pixels that decrease the transmittance, the camera operates in low- light conditions, resulting in noise amplification. The degraded images in the UDC dataset should effectively contain the unique UDC noise, which differs from standard cameras. For example,

---

(a) Zhou-S
(b) Feng-R
(c) Feng-S S
(c) UDC-SIT
GT
Degraded
GT
Degraded
GT
Degraded
GT
Degraded
Noise
Noise
Figure 5: Comparison the transmittance decrease and UDC noises. GT stands for ground truth. (a)
Zhou-S 52 (b) Feng-R R 13 (c) Feng-S [15]. (d) UDC-SIT.
(a) Zhou-S
(b) Feng-R
(c) Feng-S
(d) UDC-SIT
GT
Degraded
GT
Degraded
GT
Degraded
GT
Degraded
02
Spatially variant flares
Spatially
variant flares
(c) Zhou-S
(f) Feng-R
gg) Feng-S
(h) UDC-SIT
GT
Degraded
GT
Degraded
GT
Degraded
GT
Degraded
CAF
Figure 6: Comparison of flares. The light sources affect the flare shape. The images in (a) Zhou- S [52], (b) Feng-R [13] (c) Feng-S [15], and (d) UDC-SIT show the UDC flares by the round shape of light sources. The images in (e) Zhou-S [2]. (f) Feng-R [13, (g) Feng-S [15). and (h) UDC-SIT show the UDC flares by fluorescent light sources.
visual comparison is given in Figure 5 The image in Zhou-S in Figure 5 is captured from monitor under controlled lighting conditions. Thus, there exists no noise. However, we observe the excessive transmittance decrease caused by P-OLED. In addition, the image in Feng-S in Figure 5)c) generated by convolving the PSF on an HDR image from the HDRI Haven dataset [18]. The HDR images in the dataset are meticulously created through exposure bracketing (3] and cleanup to ensure high quality without overexposure, chromatic aberration, and noise. Thus, the Feng-S image has reduced noise and minimal transmittance loss. On the other hand, the images in Feng-R in F and in UDC-SIT Figure d) accurately depict the visible noise caused by the UDC. They also show the effect of transmittane decrease.
7

---

Table 2: The PCK comparison between the datasets. UDC-SIT shows the best alignment and has PCK values close to 100% for all values of a.
Dataset
Need alignment
PCK (a 0.01)
PCK (ax = 0.03)
PCK (a = 0.10)
Shou-S
52
98.11
98.45
99.08
Feng-S
15
99.95
99.96
99.99
Feng-R
58.75
95.08
99,93
UDC-SIT
97.26
98.56
99.35
Flares. Lens flares typically occur when intense light scatters or reflects within an optical sys- tem [TT]. In contrast, UDC flares result from light interacting with the display panel above the camera sensor, resulting in undesired reflections, diffraction, or scattering. Thus, it is essential for the images the UDC dataset exactly describe real UDC flares if they exist. For example, a visual comparison is given in Figure 6 The images in Zhou-S in Figure G(a) and (e) do not exhibit any flares since they capture images displayed on monitor. Similarly, the images in Feng-S in Figure (()) and (g) generate unrealistic flares by convolving PSFs to HDR images. Specifically, it lacks the generation of glare, streaks, and spatially variant flares. The shapes of real-world UDC flares vary depending on their positions, resulting in different diffraction patterns. However, the flares in the image in Feng-S in Figure (6(c appear overly regular. (FigureBlc) versus (b) and (d)). Moreover, the image in Feng-S in Figure (]g) demonstrates different fluorescent light flares from the real-world UDC
flares Figure 6 f) and (h). Figure bbb) versus (d) and (f) versus (h) depict various flare shapes from different UDC smartphones.
(8)
Mechanieat Propert
Meshonica
Mechanical Projert
VEN BEHAVO
META EEHAVO
WOOLL
BEHAVC
Figure 7: Invalid mask visualization in Feng-R [13]. (a) The degraded image captured by the ZTE
Axon 20 (10) UDC. (b) The ground-trune image captured by the iPhone Pro rear camera [25] (c)
Aligned ground-truth image by AlignFormer [13]
Occlusion regions. Although Feng-R consists of real images, it has some limitations. They avoid
capturing close objects to avoid parallax and occlusion. Nonetheless, AlignFormer (13) used in Feng-R creates occlusion regions marked red in Figure This area is typically excluded in training the DNN restoration models. This is why they call Feng-R. as pseudo-real pairs.
Table 3: Analyzing UDC-SIT's PCK relative to '₁ A2, Ag. and Orotation When no rotation is applied, Protation O; otherwise, rotatton = 0,. Notably, X2 significantly improves the alignment over using MSE alone (i.e., the case (1, 0, O, 0)).
(A1, A2, Ag, Protatica)
PCK (a 0.01)
PCK (ar = 0.03)
PCK (a = 0.10)
0.
0,
0
)
78.77
81.09
85.65
0,
1
0,
0
34.05
50,70
64.66
(
0,
0.
1.
0
62.27
64,59
72.54
(
1,
1
0,
35.55
49.41
63.25
0,
1,
98.22
98.73
99.31
1.
86.13
95.02
99.23
0.1.
0.1,0
72.40
77.40
83,30
0.
0.3
43.83
47.78
59.60
Alignment quality. Finally, to measure alignment quality paired images, we compare
Percentage of Correct Keypoints (PCK) using LoFTR (42) as keypoint matcher by following the methodology presented by Feng el [13]. keypoint pair is considered correctly aligned when d < x mar (,, W',, where d is the position difference between pair of matched keypoints, a is the

---

Table 4: Restoration performance for synthetic and real UDC datasets. The term Input refers to the PSNR and SSIM values between the paired degraded and ground-truth images.
Input
DISCNet [15
UDC-UNet [31]
Ufomner- T 47
ECFNet 54
SRGAN (29
F
15
PSNR SSIM
26.08
43.27
49.37
42.47
52.17
32.35
0.8561
0.9877
0,9933
0.9844
0.9958
0.9538
UDC-SIT
PSNR
21.03
26.32
27.44
27.28
28.22
24.70
SSIM
0.7330
0.8457
0.8637
0.8594
0.9002
0.8195
threshold, and H and W represent the height and width of the image. To apply a consistent alignment criterion across the datasets with different resolutions, we uniformly set max H, W) = 1024, aligning with Feng al. [13). Then the PCK for an image pair is defined by the ratio of the number of correctly aligned keypoint pairs to the total number of keypoint pairs. Table 2 illustrates the comparison result. Since Zhou-S and Feng-S consist of synthetic images, they do not require an additional alignment process, resulting in PCK values close to 100%. On the other hand, Feng-R undergoes alignment using AlignFormer the images in the paired dataset. achieves 58.75% for a = 0.01. Unlike Feng-R, UDC-SIT consistently exhibits PCK values close to 100% across all values of c.
Table 3 shows the analysis result of PCK values on average for various A1. A2. A3. and tooation combinations. We found that the optimal combination of λ1, A2. λ3, and rooation different for each image pair. The images in UDC-SIT are selected by human inspection, and the PCK values of UDC- SIT in Table ariginntt from the human assessment. The human inspection result is comparable to results obtained by the best combination (1, 0, 1, 0) in Table 3
Applying rotation (i.e., the case of (1, 0, 0, 0.3)) from Algorithm significantly reduces PCK as shown in T While this rotation causes slight MSE decrease, its impact on digitized images leads to notable PCK decline because of changes in ordinary translational correlation during the rotation (7).
To mitigate rotation and tilt effects, we use an image-capturing system in Figure 2 applying only shifts in (x, y) coordinates. Employing the DFT loss without MSE (the cases of (0. T. 0, and (0, 1. 0j) yields lower PCK values of 34.05% and 62.27%, respectively (a 0.01). Using only MSE and AFamp( (u, v) without Aø(u, v) (case (1, 1, O, 0)) results a reduced PCK of 35.55% (a 0.01). The loss functions encompassing spatial and frequency domains, focusing on 6(u, v) of F(u, v) (the cases of(1, O, I, 0) and (1, 1, I, O)), show high PCK values of 98. 98.22% and 86. 13%, respectively (a 0.01). The case of (1, 0.1, 0.1, 0) lacks sufficient X2 and Aa to reflect the frequency domain loss effectively.
Effects on learnable restoration models
We demonstrate the effectiveness and benefits of UDC-SIT by comparing the UDC image restoration performance with Feng-S. We only compare UDC-SIT to Feng-S because Yoo-S is not publicly available, Feng-R is planned for future release but is currently unavailable, and Zhou-S needs more flares that are essential for UDC image restoration. We use five saaming-bassed image restoration models, including DISCNet [5]. UDC-UNet (3). Uformer (47), ECFNet (54), SRGAN (29) using UDC-SIT. We modify the authors' code of the models to account for the difference in dynamic range and channel size between UDC-SIT and Feng-S [15). ensure reproducibility, we explain the experimental setting in the appendix and our GitHub
The restoration performance of the five models Feng-S and UDC-SIT is given in Table 4 Models trained with Feng-S successfully restore the images with up to 2.0 times higher PSNR values than Input (except SRGAN). They also achieve nearly perfect SSIM score. However, none of the models trained with UDC-SIT can restore the images to the same extent as Feng-S. Feng-S inadequately represents actual UDC degradation, such as intense flares. The high degree glares, shimmers, and streaks results in substantial information loss. Consequently, model performance degradation occurs when those obscured objects become unrecognizable. Thus, restoration becomes more challenging with UDC-SIT than with conventional synthetic datasets.
Effects of annotations. It is noteworthy that annotations are specifically available in the UDC-SIT. Flare scenes heavily rely the capturing environment. For example, artificial lights emit a different

---

(a)
(b)
(c)
Glare
Shimmer Shimmer
Streak
(d
Glare Shimmer
(e)
Streak
Glare
Glare
Streak
Streak
Shimmer
g
(hy
Figure 8: Description of flare components. (a)-(c) The components of a flare include glare, shimmer, and streaks (TT). (d)-(f) Flare components observed in actual degraded images of UDC-SIT. (g)-(i) Ground-truth images corresponding to the UDC degraded images.
spectrum than natural sunlight, resulting in a distinct diffraction pattern. Weak light sources can cause streaks under low-light conditions. Thus, attach annotations, such light source, day/night, indoor/outdoor, and flare components, to the images in UDC-SIT. UDC-SIT follows the classification labels of flare components by Dai et (TI) including shimmer, streak, glare, and light source as illustrated in Figure 8 Detailed explanations and instructions for the annotations, and experimental result of their effect the restoration models can be found in the appendix.
Limitations
The degradation of images by the UDC depends on the display pixel patterns, which vary between products, as shown Figure When obtaining UDC-SIT, Galaxy Fold 3 (39) display panel used. Thus UDC-SIT is optimal for restoring Galaxy Z Fold 3 [39] images. Models trained with UDC-SIT may not suitable for restoring UDC images taken by other devices (e.g., ZTE Axon 20 [10] and Galaxy Z Fold [40).
7 Conclusions
As far as we know, UDC-SIT is the first dataset to include real-world UDC degradation, such as low transmittance, blur, noise, and flare, along with detailed annotations for the light source, day/night, indoor/outdoor, and flare components. With UDC-SIT, one can train UDC image restoration model improve the quality of UDC images taken by the UDC. We propose an effective image-capturing system for paired UDC-distorted and ground-truth images. We also propose a technique for aligning the paired images by exploiting the discrete Fourier transform. The experimental result of comparing UDC-SIT and representative synthetic UDC dataset with five representative learnable restoration models indicates that the models trained with the synthetic UDC dataset are impractical because the synthetic UDC dataset does not reflect the actual characteristies of UDC-degraded images. This implies UDC-SIT is an adequate dataset for UDC image restoration.
Acknowledgment
This work supported in part by the Institute for Information & communications Technology Pro- motion (IITP) grant (No. 2018-0-00581, CUDA Programming Environment for FPGA Clusters) and by the National Research Foundation of Korca(NRF) grant funded by the Korea governmen((MSIT) (No. RS-2023-00222663) This work was also supported in part by the Samsung Display Co., Ltd. ICT at Seoul National University provided research facilities for this study.
10

---

References
[1] AbdeIrahman Abdelhamed, Stephen Lin, and Michael S Brown. "A high-quality denoising dataset for smartphone cameras".. In: Proceedings of the IEEE conference computer vision and pattern recognition. 2018, pp. 1692-1700.
[2] Adobe. About high dynamic range images. Available at https: https:// /tep:: adobe... com/ /
photoshop/usst http////iigghddyann hwwphhhhhhdnmmc-aaaaaaaaaaa cangeee-raannnnnnn
httml 2023.
[3] Adobe. What is bracketing in photography? Available https : / www adobe com / creativecloudd photography / hub guides / what is bracketing in photography.. html 2023.
[4] Bryce E. Bayer. Color imaging array. US Patents US3971065A, 1975.
[5] Glenn D Boreman. Modulation transfer function in optical and electro-optical systems. Vol. 4. SPIE press Bellingham, Washington, 2001.
Hantice-Hall, [6] E Oran Brigham. The fast Fourier transform and its applications. Prentice- Hall, Inc., 1988.
[7] Matthew Brown and David Lowe. "Automatic panoramic image stitching using invariant features". In: International journal of computer vision 74 (2007), pp. 59-73.
[8] Tianqi Chen et al. "Training deep nets with sublinear memory cost". In: arXiv preprint arXiv:1604 (166440617) (2016).
[9] Sung-Jin Cho et al. "Rethinking coarse-to-fine approach in single image deblurring". In: Proceedings of the IEFE/CVF international conference on computer vision. 2021, pp. 4641 4650.
[10] ZTE Corporation. ZTE Axon 20. Available at http::://zteeevicsssso eon en- 2te-
axon 20- 12-5g-speca/ 2020.
[11] Yuckun Dai et al. "Flare7K: A Phenomenological Nighttime Flare Removal Dataset". In:
Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks
Track. 2022.
[12] Yuekun Dai et al. "Nighttime Smartphone Reflective Flare Removal Using Optical Center Symmetry Prior". In: arXiv preprint arXiv:2303. 15046 (2023).
[13] Ruicheng Feng et al. "Generating Aligned Aseudd-SSuperviie
from Non-Aligned Data for Image Restoration in Under-Display Camera". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 5013-5022.
[14] Ruicheng Feng et al. "Mipi 2022 challenge on under-display camera image restoration: Meth- Aition-ECVV/i 2022 Workshops: Tel Aviv, Israel, October 23-27,
ods and results". In: Computer 2022, Proceedings, Part V. Springer. 2023, pp. 60-77.
[15] Ruicheng Feng et al. "Removing diffraction image artifacts in under-display camera via dynamic skip connection network". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 662-671.
[16] Martin A Fischler and Robert C Bolles. "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography". In: Communications of the ACM 24.6 (1981), pp. 381-395.
[17] Dario Fuoli, Lue Van Gool, and Radu Timofte. "Fourier space losses for efficient perceptual image super-resolution" In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 2360-2366
[18] Poly Haven. HDRI Haven. Available at httttss://ddriaavee. com/hdris///////cccccoooccc
2020.
[19] Kaiming He et al. "Deep residual learning for image recognition".. In: Proceedings of the IEEE
conference on computer vision and pattern recognition. 2016, pp. 770-778.
[20] Alain Hore and Djemel Ziou. "Image quality metrics: PSNR vs. SSIM". 2010 20th international conference on pattern recognition. IEFE. 2010, pp. 2366-2369.
[21] SHIN HoWon and JO Soyeon. Organic light emitting display device to implement narrow bezel and thin thickness. US Patent 10,658,436. May 2020.
[22] Xun Huang and Serge Belongie. "Arbitrary style transfer in real-time with adaptive instance normalization". In: Proceedings of the IEEE international conference on computer vision. 2017, pp. 1501-1510.
[23] Andrey Ignatov, Luc Van Gool, and Radu Timofte. "Replacing mobile camera isp with single deep learning model". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020, pp. 536-537.

---

[24] Andrey Ignatov et al. "Dslr-quality photos on mobile devices with deep convolutional net- works". In: Proceedings of the IEEE international conference on computer vision. 2017, pp. 3277-3285.
[25] Apple Inc. iPhone 13 Pro. Available at https : ://www apple com com/by iphone 13
pro/specs/ 2021.
[26] KIM Jacbok and Changhoon Jeon. Narrow bezel display apparatus using a folded substrate.
US Patent 9,760, 125. Sept. 2017.
[27]
Liming Jiang et al. "Focal frequency loss for image reconstruction and synthesis". In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 13919- 13929.
[28] Tero Karras, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 4401-4410.
[29] Christian Ledig et al. "Photo-realistic single image super-resolution using generative ad- versarial network". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 4681-4690.
[30] Shen Li. Getting Started With Distributed Data Paralle(DDDP). 2023. URL: https / pytorch... wtw.tootrggttttteeeeeee instorials/inne httpaaaaaaaaatemmmmaaaaaac..pppp
.ttm (visited on 06/03/2023). [31] Xina Liu et al. "UDC-UNet: Under- Display Camera Image Restoration via U-shape Dynamic Network". In: Computer Vision ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. Springer. 2023, pp. 113-129.
[32] David G Lowe. "Distinetive image features from scale-invariant keypoints". In: International journal of computer vision 60 (2004), pp. 91-110.
[33] Rafal Mantiuk et al. "High dynamic range image and video compression-fidelly matching human visual performance". In: 2007 IEEE International Conference on Image Processing.
Vol. 1. IEEE. 2007, pp. 1-9.
[34] Tobias Plotz and Stefan Roth. "Benchmarking denoising algorithms with real photographs". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 1586-1595.
[35] Zong Qin al. "See-through image blurring of transparent organic light-emitting diodes display: calculation method based on diffraction and analysis of pixel structures". In: Journal of Display Technology 12.11 (2016), pp. 1249..
[36] Erik Reinhard al. "Photographic tone reproduction for digital images". In: Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 2002, pp. 267-276.
[37] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation". In: Medical Image Computing and Computer-Assive Intervention-MiiCAA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings. Part III 18. Springer. 2015, pp. 234-241.
[38]
Ltd. Samsung Electronics Co. Samsung Galaxy Note 10. Available at attttps:: battps://www. http:://wwwss.muuggggg.o
com/my http://wwaaaartopooe/o/laaaa/aes
2019.
[39]
Ltd. Samsung Elletroncc Co. Samsung Galaxy 3. Available at
samsung
com/ com/ http:///aa//lllleelllllayyer////
2021.
[40]
Ltd. Samsung Electronnes Co. Sww.snwwwwnnnnnaaaaaaaaaaaaaaaaa Galaxy Available atttttps:/www.aamtt 2022.
com com/global wttp:///aaa/llaaaag/gaaa////////
[41]
KIM SangWoon et al. Narnow bezel display device. US Patent 10,114,258 Oct 2018.
[42]
Jiaming Sun et al. "LoFTR: Detector-free local feature matching with transformers". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021,
pp.
[43] Qilin Sun et al. "Learning rank-1 diffractive optics for single-shot high dynamic range imaging". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020, pp. 1386-1396.
[44] Varun Sundar et al. "Deep atrous guided filter for image restoration in under display cameras". In: Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings. Part V 16. Springer. 2020, pp. 379-397.
12

---

[45] Zachary Teed and Jia Deng. "Raft: Recurrent all-pairs field transforms for optical flow". In: 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. Springer. 2020, pp. 402-419.
Computer
[46] Shitong Wang al. "Distortion correction in surface defects evaluating system of large fine optics". In: Optics Communications 312 (2014), pp. 110-116.
[47] Zhendong Wang et al. "Uformer: A general u-shaped transformer for image restoration". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022,
Pp. 17683-17693.
[48] Yicheng Wu et al. "How to train neural networks for flare removal". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 2239-2247.
[49] Qirui Yang et al. "Residual and dense unet for under-display camera restoration". In: Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part Springer. 2021, pp. 398-408.
[50] Jewon Yoo et al. "7-5: Synthetic Dataset for Improving UDC Video Restoration Network Performance". In: SID Symposium Digest of Technical Papers. Wiley Online Library. 2022,
pp. 58-60.
[51] Syed Waqas Zamir et al. "Restormer: Efficient transformer for high-resolution image restora- tion". In: Proceedings of the IEEEJCVF Conference on Computer Vision and Puttern Recogni- tion. 2022, pp. 5728-5739.
[52] Yuqian Zhou et "Image restoration for disera camera". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 9179-9188
[53] Yuqian Zhou et al. "UDC 2020 challenge image restoration of under-display camera: Methods and results". In: Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16. Springer. 2020, pp. 337-351.
[54] Yurui Zhu al. "Enhanced coarse-to-fine network for image restoration from under-display cameras". In: Computer Vision- ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. Springer. 2023, pp. 130-146.
13

---

A Background
In this section, discuss the factors that contribute the UDC degradation. We also explore the concept of dynamic range and tone mapping. Also, we discuss color representation and image formats to elucidate the rationale behind collecting the UDC-SIT dataset the form RAW images with an GREG 4-channel Bayer pattern.
A.1 Image degradation due the UDC
The representative reasons for image degradation due to the UDC can be categorized into three classes: noise, low transmittance, and flare caused by the display panel top of the camera lens. The reasons for the degradation differ depending on various factors, such as display types (T-OLED and P-OLED), pixel designs, the distance from the display to the camera lens, and the lens structure. Qin et al. (35] demonstrates that pixel designs affect the diffraction level. The optimal pixel design could improve the image quality of the UDC. However, the design of the pixels needs to consider many factors, not only the degradation by the UDC but also the size and shape of the red, green, and blue sub sub-pixels. They affect the color intensity and image sharpness.
Under the UDC setting, because the pixels work as if they were slits, as shown in Figure A.I the light diffracts, and this causes the degradation. The light diffracts when it passes through obstacles comparable in size to wavelength. Unfortunately, we cannot avoid diffraction because the aperture size of the pixel layout is on the order of the wavelength of visible light. The diffracted light propagates the camera sensor through the camera lens.
The Point Spread Function (PSF) of an optical system is the image-irradiance distribution or impulse response on an image plane that results from point source (5). For example, an image of distant star through telescope is a PSF It is an energy distribution pattern. It peaks in the center and has characteristics of long-tail low-energy sidelobes. Since different light source has different PSF, the flare shape might differ depending on the light source. Also, the diffraction levels of red, green, and blue waves differ because of their wavelengths in the visible ray region.
A.2 Dynamic ranges and tone mapping
An image's dynamic range describes a scene's luminance range or the limits of the luminance range that a digital camera can capture (33). The digital camera sensor narrower than the range of brightness that the human eye can accommodate. Multiple frames of the same scene captured with different exposures, shutter speeds, and F-numbers can be combined to create an image with a higher dynamic range than individually captured frames.
Tone mapping required to view video image captured with High Dynamic Range (HDR) monitor. Since the display device has limited dynamic range unsuitable for reproducing the full range of light intensities in an actual scene, tone mapping approximates the appearance an HDR image in a medium with a more limited dynamic range by mapping one set of colors to another. One of the representative tone mapping operations is Reinhard tone mapping [36].
(a)
(b)
Camera lens
PSF
Pixel pattern
Camera
Diffraction
PSF
sensor
Figure A.1: (a) The diffraction, (b) The PSF is attributed to the pixel pattern on the UDC area.
14

---

A.3 Color representation and image formats
4 is Color Filter Array (CFA) for arranging RGB color filters on a square grid of photosensors. uses checkerboard arrangement of filter patterns, such as BGGR, RGBG, GRBG, or RGGB. Image sensors for digital cameras typically use the Bayer pattern. Since each pixel provides only a single-color pixel value (e.g., green) obtained from CFA sensor, demosaicing is used to reconstruct the full-color RGB image. Interpolation computes the pixel's missing color values (e.g.. red and blue) using the colors of neighboring pixels.
Bayer pattern
RAW image files contain unprocessed or minimally processed data from a digital camera's image sensor. RAW image files in recent smartphone camera sensors have 10-bit range, and each pixel records one of the three colors: red, green, and blue in 1,024 steps using the Bayer pattern. Other image formats, such JPEG and PNG, converted from the RAW format using an Image Signal Processor (ISP) with some loss of information in 256 steps. Thus, collecting the images in the RAW format is essential to ensure that the dataset is free from the influence the ISP (i.e., free from the loss of information).
B Details of the UDC datasets and DNN models used
This section provides detailed information the UDC datasets and the DNN models used compare the datasets in the paper for reproducibility.
The datasheets for datasets describe UDC-SIT's hosting, licensing, and maintenance plan. The UDC-SIT dataset, code, and evaluation procedure can be found and downloaded at: https///gttab..co//mc///DDCCCCCD
B.1 UDC datasets
The images in UDC-SIT captured in the RAW format using the GRBG 4-channel Bayer pattern, effectively eliminating the influence of the built-in ISP. Although HDR captures the details shadows and highlights better, actual UDC images are predominantly in LDR. Consequently, we capture images LDR to align with real-world conditions.
Unlike Table it emphasizes information relevant the DNN model execution. Zhou et al. [52 also obtain 16-bit RAW sensor data but only release the paired RGB data. Thus, we classify Zhou-S as 3-channel RGB images. When working with HDR images, tone mapping becomes necessary. Feng el al. capture 330 images with dimensions 3200 x 2400 3. These are then cropped into 6,521 1 training images of size 512 x 512 x 3 and 226 test images sized 1024 1024 x 3. We provide the dataset in the NPY format (.npy) to make UDC-SIT open and widely used.
We compare the available UDC dataset in Table B.
B.2 DNN models for evaluating UDC-SIT
The DNN models used for evaluating UDC-SIT include DISCNet [15]. UDC-UNet 31I, Uformer- T [47.. ECFNet (54), and SRGAN (29). While we primarily adhere to the original authors' code for the models, we make some modifications.
Table B.l: Comparison of the available UDC datasets.
Zhou-S (22
Feng-S
[15]
Feng-R R [13
UDC-SIT
Raw sensor data Color representation
RGB [1024, 2048, 3]
RGB [800, 800, 3]
RGB
GRBG Bayer [1792, 1280, 4]
Dimensions
[512, 512, 3], [1024, 1024, 3] HDR V
Dynamic range
LDR
HDR
LDR
Tone mapping
V
Value range
[0, 255] .png
[0, 500] .npy
[0, 255]
[0. 1023]
File format
.png
.mpy
15

---

DISCNet, UNC-UNet, and ECFNet are designed to restore UDC images in HDR (i.e., Feng-S (T5]). while Uformer-T designed to restore motion-blurres images in LDR. Also, SRGAN is designed to super-resolve at large upscaling factors while also taking into account the preservation of high- frequency details. Consequently, we make some necessary adjustments the PyTorch Dataloader, implementing tone mapping [36] for Feng-S and opting for normalization instead of tone mapping for UDC-SIT. Also, we increase the channel size the models to to match the channel size of UDC-SIT. Training is performed with four NVIDIA GeForce RTX 3090 GPUs. The following are the training details specific to each model for the UDC-SIT dataset:
DISCNet. Due to the channel size of 4 in UDC-SIT, we solely utilize I.1 loss as the loss function, excluding perceptual loss. To accelerate training, the models are trained using PyTorrch Distributed Data Parallel (DDP) (30) with a batch size of 32.
UDC-UNet. The model output is clamped to the range [0, 1] to calculate the training loss. The additional modification aligns with the details described in DISCNet.
Ufomee-T. To maintain consistency with other models operating solely in FP32, we have modified Uformer-T T to use FP32 instead of mixed precision. Also, we utilize Transformer The patch size is set to 1, 280 x 1, 280, and the batch size is
feature channels with C
ECFNet. Since the training script of ECFNet is not publicly available, we follow the method used in MIMOUNet (9] that heavily affected ECFNet. We increase the base channel size of blocks from 24 to 32 to provide eight channels per image channel. ECFNet introduces the progressive training strategy adopted by Restormer (5T). Specifically, the training phase of the network consists of three stages with different patch sizes (e.g., 256 x 256, 512 x 512, and 800 x 800). We modify the batch size each stage from 3, 1, and to 16, 4, and 4, respectively. In the third stage, we employ activation recomputation (8] to address the memory constraint of NVIDIA GeForce RTX 3090 GPUs. effectively reduces the peak memory usage from 46GB to 18GB per device, enabling efficient utilization of available GPU memory resources. We do not use external data or adopt model ensemble strategies that the authors of the models describe.
SRGAN. The model output is clamped to the range [0, 1] to calculate the training loss. Due to the channel size of 4 in UDC-SIT, utilize L2 loss and adversarial loss the loss function, excluding perceptual loss.
C Details of the annotations
This section describes annotation instructions for crowdsourcing and the effect of annotations in the UDC-SIT.
Table B.2: Annotation examples for UDC-SIT. The corresponding annotated images are shown in Figure B.2 "Indoor" and "Outdoor" scenes are labeled 1 and 3, respectively. The labels for "No distinction." "Day." and "Night" are 1, 2, and 3, respectively. For indoor images, if flares occur due to the influence of natural light sources through windows, we label them "Day" and "Night". For indoor scenes without windows, we label them "No distinction". Glares, shimmers, and streaks are labeled I present and 0 otherwise. If there is no flare, labeled 0. The occurrences of flares caused by natural sunlight, artificial light, both are labeled 1, 2, and 3, respectively.
File
Indoor/Outdoor (1/3)
distintiion/Day/Nihh Glare (0/1)
Shimmer (0/1)
Streak (0/1)
Light sources
(1/2/3)
(0/1/2/3)
317.npy
3
3
2
333.npy
3
2
677.npy
3
2
0
689.npy
3
1
698.npy
1
903.ngy
905.npy
2384.npy
3
16

---

(
PSNR 20.15
PSNR 23.43
PSNR 26,09
PSNR
26.18
PSNR
23.4
PSNR
23 42
SSIM 0.7823
SSIM 0.8894 SSIM 0:9079
SSIM
0,9016
SSIM
0.9134
SSIM
9 8638
(b)
PSNR 12.78 PSNR 30.31
12.
SSIM -0.7710
SSIM 0.8776 SSIM 9079 SSIM
0,8942
SSIM 0.93 04.SSSSSMM 0,8541
(c)
PSNN 16.86
PSNR 14.91
PSNR 15.64 PSNR 14,90 PSNR 16.13 PSNR 14.62
SSIM 0.6412
SSIM 0.6514 SSIM 0.68599 SSIM 0.6880 SSIM O. 6944SSIM 0.6677
(d)
PSNR 15.59
PSNK 25.39
PSNK 28.51 PSNK 25.30
PSNK
28 PSNK 20:42
SSIM 0.6070
SSIM 0.7896 SSIM 0.8171 SSIM 0.7913
SSIM 0.8985
SSIM 0.7104
Input
GT
DISCNet UDC-UNet Uformer-T
ECFNet
SRGAN
Figure B.I: Visual comparison of flare removal on real-world UDC images. Flare shapes vary according to the captured environments. GT stands for ground truth. (a) Indoor artificial light (glare + shimmer). (b) Indoor + artificial light (glare shimmer + streak). (c) Outdoor + day + sunlight (glare shimmer streak). (d) Outdoor night artificial light (glare shimmer streak).
Table B.3: Annotation distribution and the number of pairs.
Label
# of pairs
Indoor (1)
1,754 586
Outdoor (3)
No distinction (1)
1,340
Day (2)
649
Night (3)
351
Glare (0 1)
2,037
Shimmer (0 or 1)
1,899
Streak (0 or 1)
1,067
No Haaee (0)
273
Natural light (1)
175
Artificial light (2)
1,639
Both (3)
253
Total
2,340
C.1 Effects of image capturing environments
The combination of various factors, including light sources, indoor/outdoor, and day/night, affect the level of degradation flare shapes in UDC images. The images in Figure|B.I] [[T show only glare and shimmer, while the images in Figure B. B.I display glare, shimmer, and streak caused by different artificial light sources. Natural sunlight Figure [c) has the strongest intensity. shows extreme flares with glare, shimmer, and streak even in the daytime, while artificial light at night with the low-light condition Figure dd) has relatively weak intensity but still has extreme glare, shimmer, and streak.
17

---

C.2 Types of annotations
We offer annotations for each image pair. Table provides a detailed overview of the total count and distribution of different annotation labels. Note that an image pair can have multiple annotation labels. The parenthesized number beside a label is the encoding of the label. The pairs are categorized based on image degradation factors, such as indoor/outdoor, day/night, glare/shimmer/streaak and light sources. If images captured indoors without any window access to natural sunlight, there would be no distinction between daytime and nighttime. In such cases, we label them as "No distinction." Detailed information can be found at: http:///ttub.ooo//////CCCCCCC//
Participants involved the crowdsourcing for UDC-SIT are provided with the annotation examples in Table B.2 The corresponding annotated images are shown in Figure B.2 The details of the annotations are described as follows.
Flare components. We follow the classification scheme proposed by Dai et al. 11 to categorize different flare components, including glares, shimmers, and streaks. Figure in the main body each flare
of the paper and Figure B.2 provide detailed explanations and visual representations component, serving as comprehensive annotation instructions. A glare refers the presence of intense and strong light that produces artifacts such as circular light patterns. A shimmer involves rapid and subtle light or color intensity variations within an image. streak long, thin, and
(a)
Glare
(b)
Streak
Shimmer
Shimmer
Glan
(c)
(d)
Glare Shimmer
Streak
(e)
Shimmer
(f)
Shimmer
Glaam
(g)
(h)
Clare Shimmer
GT
Degraded
GT
Degraded
Figure B.2: UDC-SIT in various scenarios. Table B shows the corresponding annotations. (a)
317.npy. (b) 333.npy. (c) 677 677.npy. (d) 689.mm .py (e) 698.npy. (f) 903.npy. (g) 905.npy. (h) 2384.npy.
18

---

Table C.I: The comparison of restoration performance of the DNN models regarding the annotations UDC-SIT. "Input" refers to the PSNR and SSIM between the degraded and ground-truth images in
UDC-SIT.
Inpul
DISCNet [15
UDC-UNel 31
Ufommer-T 47
BCFN 54
SRGAN 29
Indoor
PSNR
21.17
26.63
27.79
27.58
28.64
25.06
SSIM
0.7437
0.8535
0.8715
0.8680
0.9058
0.8311
Outdoor
20.53
25.18
26.13
26.15
26.66
23.37
SSIM
0.6924
0.8163
0.8346
0.8272
0.8795
0.7761
Day
PSNR
21.54
26.23
27.28
27.47
28.17
24.67
SSIM
0.7689
0.8669
0.8857
0.8823
0.9152
0.8408
Night
PSNR
19.63
25.32
26.62
25.99
27.06
23.62
SSIM
0.6012
0.7436
0.7633
0.7536
0.8485
0.7016
Sunlight
PSNR
20.48
24.49
25.52
25.62
26.22
22.72
SSIM
0.7488
0.8475
0,8670
0.8637
0.8966
0.8147
Artificial light
PSNR
20.37
26.40
27.67
27.19
28.37
24.72
SSIM
0.7026
0.8279
0.8459
0.8401
0.8895
0,8022
No light
PSNR
22.43
26.22
26.52
27.18
27.52
24.95
SSIM
0.7180
0.8521
0.8645
0.8623
0.8893
0.8103
Glare
PSNR
20,91
26.37
27.57
27.29
28.32
24.71
SSIM
0,7351
0,8456
0,8643
0,8598
0,9024
0,8211
Shimmer
PSNR
20.67
26.41
27.62
27.33
28.42
24.73
SSIM
0.7301
0.8413
0.8601
0.8557
0.9000
0.8181
Streak
PSNR
26.05
27.27
26.83
28.04
SSIM 0.7109
0.8266
0.8465
0.8400
0.8897
0.8026
typically irregular line of light or color observed in image. Note that the shapes of glares and shimmers vary depending on factors such as day and night conditions, indoor and outdoor settings, and light sources, as illustrated in Figure B.2 In addition, Figure B.2 demonstrates that the shape and intensity of glares, shimmers, and streaks differ based on the location within the scene, even when originating from the same light source.
Light sources. The intensity and shape of flares vary also depending on the light sources. Flares generated by natural sunlight, depicted in Figure B.2 exhibit extreme intensity and often include streaks. Additionally, flares can have different characteristics under different conditions, such as day/night and indoor/outdoor settings. For example, the flare in Figure [B.2,e) is also caused by natural sunlight, but its shape differs from the flare in Figure B.2 Also, Figure B.2 a), Figure B.2 (b), Figure B.2 f), and Figure B.2 g) demonstrate variations the shape of flares caused by different types of lights. Particularly, Figure (b) demonstrates that artificial lights in close proximity during nighttime
induce streaks, and Figure B.2 gg) highlights how different types of artificial lights result in various flare characteristics.
Day/Night. In the case of indoor images, such those depicted in Figure B.2 g). the distinction between day and night is generally not required (labeled as "No distinction" in Table B.2 However, we distinguish between day and night when sunlight enters the indoor space through a window, leading to flare occurrence as shown in Figure B.22 Indoor images with windows, even if no flare occurs (Figure (f),)), are still labeled day night. This reflects that windows indoor spaces may not always cause flares during the day, as flare occurrence can depend on the presence and intensity of sunlight between day and night.
The illuminance outdoors is generally higher than that indoors during the daytime. Figure (32 and Figure B.2 e) demonstrate that the flares caused by sunlight exhibit variations de- pending on whether the image is captured indoors outdoors. Interestingly, the flares in Figure B.2 are caused by the reflection of sunlight on the exterior walls of the building.
Indoor/Outdoor.
C.3 Restoration performance regarding the annotations
Table C.I
shows the comparative analysis of the restoration performance achieved by different DNN models depending on UDC-SIT annotations. Since a UDC-SIT image generally has multiple annotations, the annotation type in a row cannot be regarded as the sole factor influencing UDC degradation when obtaining the PSNR and SSIM values. However, it reasonable acknowledge the annotation type significant factor that affects the PSNR and SSIM values.
19

---

Overall, there is tendency for the restored image to have low PSNR and SSIM values when the input pair has low PSNR and SSIM values. For instance, Table C. demonstrates that input pair captured in outdoor environments exhibit lower PSNR and SSIM values than indoor images. In turn, the restored images by the models also demonstrate inferior PSNR and SSIM values for outdoor restorations compared to indoor restorations. This observation can be attributed to strong sunlight, and intense flares often accompanying outdoor scenes.
The low-light conditions at night influence the degradation significantly. Night conditions tend to be darker, increasing noise and introducing extreme flares. This phenomenon can be observed from the lower PSNR and SSIM values for the annotation Night compared to Day, as indicated in Table C.1
The presence of flares and the type of light sources inducing flares also significant factors in UDC degradation. Table C.1 demonstrates the impact of flares on degradation. The input images degraded by sunlight and restored images have low PSNR values compared the no-light condition. However, interestingly, restored images from the images degraded by artificial light depict higher PSNR values compared the no-light condition. This tells us that the DNN models are relatively proficient in restoring the degradation caused by artificial light compared to natural sunlight.
Under the sunlight condition, the SSIM values are higher than under the no-light condition. On the contrary, the SSIM values under the artificial light condition are less than those under the no-light condition. PSNR measures the noise level by comparing the original and degraded images. Increasing degradation leads to higher noise and lower PSNR values. On the other hand, SSIM measures structural similarity, considering brightness, contrast, and structural features. Thus, SSIM may exhibit slightly different sensitivity towards different types of degradation compared to PSNR (20). degradation factors like flares do not significantly reduce structural similarity, SSIM values will be relatively high.
When flares are present in degraded image, glares and shimmers appear over wider area than streaks. Glares and shimmers may significantly impact the reduction of PSNR and SSIM values. On the other hand, streaks typically occur in conjunction with glares and shimmers rather than standalone phenomenon. Thus, there may not be significant differences in PSNR and SSIM values based glares, shimmers, or streaks individually, shown in Table C.1 However, low-intensity artificial lights at night or natural sunlight in the daytime may cause extreme flares, including streaks. These extreme flares introduce more significant degradation than glares shimmers alone. Thus, streaks generally exhibit slightly low PSNR and SSIM values compared to glares and shimmers due to their association with extreme flares, as illustrated in Table C.
High-resolution images taken by modern smartphones often require training the image-restoration DNN models by patching smaller images. The annotated information, such as indoor/outdoor, day/night, glare/shimmmerrstrakk,, and light sources, attached to each patch facilitate the patch-level training in UDC restoration tasks. For example, restoring streaks is challenging for the DNN models when performing the patch-level training, as illustrated in Figure [.T.c). Figure B () shows big streaks that cover almost the entire image. Providing the annotation information about the entire image can enhance patch-level training by enabling the model to learn correlations between patches and patterns across the patches. This additional information improves the model's 's ability restore enhance images effectively.
In summary, UDC-SIT annotations are crucial for advancing UDC research none of the existing UDC datasets provide similar information.
20


Reviews:
[{"review_id": "aIEMi7kI81", "reviewer": "Reviewer_YpcN", "summary": "", "questions": "", "limitations": "1. It seems that the additional annotations including the type of flares, the light sources are not utilized in this paper. Can authors provide some simple applications with that part of annotations to show its usage and importance compared to other datasets?\n\n2. The proposed datasets is only applicable to specific display type and camera using to collect such a dataset. It is recommended that some generalization experiments are conducted to simply show such issues and give some insights on what needs to be handled in the future.\n\n3. UDC is usually used as the front-camera including smart phones and monitors, as motivated in this paper, is there any plans to include the  selfie scenarios or test on such cases while keeping the privacy unvoilated?", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. A new real-world UDC degradation dataset with aligned images.\n2. The acquistion equipment is easy yet effective.\n3. The alignment technique seems useful for aligning the images with/without UDC degradation.\n4. The comparison with prior datasets are detailed and the paper is easy to follow.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mV0Pdukj1u", "reviewer": "Reviewer_iqzz", "summary": "", "questions": "", "limitations": "Limitations are discussed in this paper.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper has a good literature overview of the existing UDC image restoration dataset, provides a comprehensive analysis of existing UDC datasets, and identifies their limitations. The authors devise and propose an image-capturing system to obtain matched pairs of undistorted and UDC-distorted images. Small geometric disparities are further eliminated by the proposed image aligning technique using Discrete Fourier Transform. The proposed real-world UDC dataset, UDC-SIT, accurately reflects actual image degradations by UDC technology, with comprehensive annotations. Comparison of UDC-SIT with a synthetic UDC dataset using four representative learnable image restoration models to demonstrate its effectiveness. The dataset has the potential to inspire further research in the community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "jE7tKy9UKJ", "reviewer": "Reviewer_6C5X", "summary": "", "questions": "", "limitations": "1. Four representative learnable image restoration models are not enough. How about GAN-based network, diffusion-based network? I am wondering how they performed compared to transformer-based networks (Uformer).\n\n2. Discrete Fourier transform seems used a lot in image restoration, including UDC. I am wondering is there any differences with the proposed paper and other existing methods.\n\n3. Will the atmosphere also influence the UDC? For example, UDC image under foggy days, will the atmosphere scatter influence lens flare, and make the lens flare more obvious? Have the dataset consider this kind of flare?", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. It is interesting to see Under Display Camera (UDC) combine the Flare7k dataset and do the annotation.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "QzYEJKOFiO", "reviewer": "Reviewer_7scG", "summary": "", "questions": "", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The data collection method is very reasonable.\n2. The image alignment technique is novel.\n3. The dataset is very challenging for the existing image restoration methods.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper introduces a real UDC dataset, which is achieved by an image-capturing system and an image alignment technique. This dataset holds significant importance in the research area. Additionally, the system and alignment methods introduced in the paper offer valuable insights for related research efforts. The experiments conducted in the study are thorough and sufficient.

The consensus among the four reviewers (6,6,6,6) resulted in an average rating of 6 across the board. The primary concerns raised by the reviewers primarily revolved around details, most of which have been adequately addressed by the authors in the updated version and subsequent discussions. While the paper received an average score, the general consensus was that it was marginally above acceptance threshold. Furthermore, the concerns and minor issues could be resolved in the final version of the paper.

Therefore, the AC recommends the paper for acceptance. The feedback provided by the reviewers is valuable and can be used to improve the paper further. It is essential to incorporate all modifications and updates into the final version and ensure consistency in the referencing style, taking into account that some papers have been accepted by journals and conferences, not just in their arXiv versions.
Decision: Accept (Poster)