SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Recaptured Raw Screen Image and Video Demoiring via Channel and Spatial Modulations
Huanjing Yue
huanjing.yue@tju.edu.cn
Yijia Cheng
yijia_cheng@tju.edu.cn
Xin Liu
Lappeenranta-Lahti University of Technology LUT
Lappeenranta, Finland
linuxsino@gmail.com
Jingyu Yang *
yjy@tju.edu.cn
Capturing screen contents by smartphone cameras has become a common way for information sharing. However, these images and videos are often degraded by moiré patterns, which are caused by frequency aliasing between the camera filter array and digital display grids. We observe that the moiré patterns in raw domain is simpler than those in sRGB domain, and the moiré patterns in raw color channels have different properties. Therefore, we propose an image and video demoiring network tailored for raw inputs. We introduce a color-separated feature branch, and it is fused with the traditional feature-mixed branch via channel and spatial modulations. Specifically, the channel modulation utilizes modulated color-separated features to enhance the color-mixed features. The spatial modulation utilizes the feature with large receptive field to modulate the feature with small receptive field. In addition, we build the first well-aligned raw video demoiring (RawVDemoiré) dataset and propose an efficient temporal alignment method by inserting alternating patterns. Experiments demonstrate that our method achieves state-of-the-art performance for both image and video demoiring. We have released the code and dataset in https://github.com/tju-chengyijia/VD_raw.
With the popularity of mobile phone cameras, the way of information transmission has gradually evolved from text to images and videos. However, when capture the contents displayed on digital screens, the recaptured images and videos are heavily influenced by the moiré patterns (i.e., colored stripes, as shown in Fig. 1) due to the frequency aliasing, and the visual contrast are also degraded. Therefore, the task of image and video demoiring is emerging.
In recent years, image demoiréing has attained growing attention and the demoiréing performance has been greatly improved with the development of deep learning networks and the constructed paired demoiréing datasets. Since the moiré patterns are of various scales and is distinguishable in frequency domain, they are removed by constructing multi-scale fusion networks [19; 6; 28; 14], frequency
*Corresponding author. This work was supported in part by the National Natural Science Foundation of China under Grant 62072331, Grant 62231018, and Grant 62171309.
Moiré full images
Figure 1: Visualization of moiré patterns in different color channels and their corresponding DCT spectra. For the sRGB channel, we only present the DCT spectrum of its intensity (Y) channel.
sRGB
R-channel
G1-channel
G2-channel
B-channel
domain modeling [11; 41; 42], or moiré pattern classification [5]. These works all perform demoiring in standard RGB (sRGB) domain. In these works, the moiré patterns are not only caused by frequency aliasing between the camera's color filter array (CFA) and the grids of display screen, but also by the demosaicing (interpolating new pixels, further aggravating frequency aliasing) algorithms in the image signal processing (ISP) [34]. In contrast, the moiré patterns in raw domain is simpler, the raw data contains the most original information, and processing moiré patterns in raw domain is more consistent with the imaging pipeline. Therefore, Yue et al. proposed demoiring in raw domain [34] and demonstrated the superiority of the raw domain processing over sRGB domain processing. However, they directly processed the raw input with an encoder-decoder network, without exploring the special properties of moiré patterns in raw domain. Therefore, developing a tailored demoiring method for raw inputs is demanded.
Compared with image demoiréing, video demoiréing is rarely studied. Recently, Dai et al. [3] proposed the first video demoiréing network with a temporal consistency loss and constructed a video demoiréing dataset. However, their dataset is constructed with one camera and one screen, which limits the diversity of moiré patterns. In addition, there are no raw videos in this dataset. Meanwhile, we observe that raw video processing is emerging in other low-level vision tasks, such as raw video enhancement [1], denoising [31; 23], and super-resolution [13; 37]. Therefore, developing a raw video demoiréing method is also demanded.
We observe that the moiré patterns in different color channels of the raw inputs have different properties. Fig. 1 presents the moiré patterns from two frames. For each image, the top row visualizes the moiré patterns in different color channels of the zoomed in patches and the bottom row presents the DCT (Discrete Cosine Transform) spectra of corresponding channels. The patches are normalized for better observation. It can be observed that the moiré patterns in sRGB domain have complex colors and the scales are different. Meanwhile, the moiré patterns in its raw RGB channels have different intensities and stripes. For example, for the first image, the moiré patterns in B-channel are more heavier while for the second image, the R-channel has more moiré patterns. This also leads to different DCT spectra, where the bright spots represent the frequencies of moiré patterns (more analysis is provided in the appendix). However, this phenomena does not exist in the sRGB domain since the demosaicing operation mixes these moiré patterns and the green channel always has the least moiré pattern due to large sampling rate. Therefore, how to utilize the above properties for raw image (video) demoiring is important. In this work, we utilize different weights to modulate the color-separated features from the four color channels. In addition, the period of moiré patterns may be large. Therefore, the features with a large receptive field can modulate the features with a small receptive field. Based on the above observations, we propose to perform raw image and video demoiring via channel and spatial modulations. Our contributions are as follows.
First, considering that the traditional cross-channel convolution will enhance the shared moiré patterns in different channels, we propose to utilize two branches. One branch is cross-channel convolution for shared feature enhancement and the other is color-group convolution with learnable parameters modulating
different color groups. Fusing the two branch features together can help remove moiré patterns and recover shared visual structures.
Second, the moiré patterns usually have a large spatial period. Therefore, we propose to utilize features with large receptive field (realized by depth-wise convolution) to modulate the features with small receptive field. In this way, we can use small computation cost to take advantage of the correlations in a large receptive field.
Third, we construct the first real raw video demoiring dataset with temporal and spatial aligned pairs. Since the frame rate of raw video capturing is unstable, we propose an efficient temporal alignment method by inserting alternating patterns. Experimental results demonstrate that our method achieves the best results on both raw video and image demoiring datasets.
2.1 Image and Video Demoiring
Compared with traditional demoiréing methods [17; 18; 26; 9; 27], the convolutional neural network (CNN) based image demoiréing methods have achieved much better performance, especially for the recaptured screen image demoiréing. Since the moiré stripes usually have different scales, many methods propose to fuse the features from different scales together [19; 6; 28; 14] to erase both large scale and small scale moiré stripes. Meanwhile, some methods propose to model the moiré patterns in frequency domain, such as wavelet domain filtering [11] and DCT domain filtering [41; 42]. Wang et al. combined the two strategies together by developing a coarse-to-fine dmoiréing network. There are also methods trying to use the classification priors. For example, MopNet [5] removed moiré patterns according to the classification of moiré stripes while Yue et al. proposed to remove moiré patterns according to the content classification priors [34]. Liu et al. proposed utilize unfocused image to help demoiréing of focused image [12]. Many demoiréing methods also emerged in famous competitions, such as the AIM challenge [30] in ICCV 2019 and NTIRE challenge [29] in CVPR 2020. Real-time demoiréing has also been explored [40]. With the development of transformer, a U-Shaped transformer was proposed to remove moiré patterns [22]. Besides supervised demoireing, unsupervised demoiréing is also explored [33]. All these methods are proposed for sRGB image demoiréing except [34], which verified that raw domain demoiréing is more appealing.
Compared with image demoiréing, video demoiréing was rarely studied. Dai et al. proposed a video demoiréing method with a relation-based temporal consistency loss [3]. They also constructed a moiré video dataset for supervised learning. However, this dataset only contains one camera and one screen, which limits the diversity of moiré patterns.
Different from them, we propose a raw image and video demoiring method by taking advantage of the moiré differences among different color channels in raw domain.
2.2 Raw Image and Video Processing
Since the raw data is linear with imaging intensity and contains the most original information with wider bit depth, many low-level vision tasks have been explored in raw domain. The representative works include raw image and video denoising [10; 21; 31; 23; 32; 4], raw image and video superresolution [39; 13; 37], raw image and video enhancement [2; 1], and raw image and video HDR [43; 36], etc. These works have demonstrated that raw domain processing can generate better results than sRGB domain processing. For demoiréing, the raw domain processing is lagging behind, especially for raw video demoiréing. In addition, the existed raw demoiréing method [34] did not tailor the networks according to the properties of raw inputs. In this work, we propose to solve image and video demoiréing in raw domain by utilizing the moiré differences in raw domain.
2.3 Demoiring Datasets
The demoiréing performance heavily depends on the training set. Therefore, many datasets for moiré removal have been proposed, which have become essential resources for researchers to train and evaluate their algorithms. The datasets in sRGB domain include synthetic datasets, i.e., LCDmoire [30] and CFAmoire [29], and real datasets, such as TIP2018 [19], FHDMI [6] and MRBI [35]. Raw
Figure 2: Visualization of the temporal confusion problem during recapturing. (a) is the ideal recapturing situation. (b) shows the actual situation, where the video displaying and recapturing do not have the same start timestamps and the raw domain recording has unstable frame rates. The temporal confusion frames are marked with red boxes.
image demoiring dataset has also been proposed [34]. During dataset construction, they usually design sophisticated methods to make the recaptured image and the source image be spatial aligned.
Compared with image dataset construction, it is more difficult to construct video demoiréing dataset since it is difficult to keep the displaying and recording be synchronized, which may lead to multi-frame confusion. Dai et al. constructed the first video demoiréing dataset [3] but there are many confused frames in their dataset. In addition, there is no raw videos in this dataset. Therefore, in this work, we construct the first raw video demoiréing dataset by inserting alternating patterns to help select temporal aligned frames from the source and recaptured videos.
3 Raw Video Demoiring Dataset Construction
Although there are many image demoiréing datasets, there's few video demoiréing dataset. In this work, we construct the first raw video demoiréing dataset by capturing the screen with handheld cameras. We observe that directly capturing the videos displayed on the screen cannot keep synchronization between the displayed frame and recaptured frame due to two problems. First, we cannot exactly make the recording and displaying start timestamps be the same. Second, the camera recording frame rates and video displaying frame rates cannot be exactly matched. Especially for raw video capturing, the recording frame rates are not stable due to the storage speed limits. For example, when we set the recording frame rate to 30 fps, the real recording frame rate varies from 27 to 30 fps. The two problems both lead to multi-frame confusion, as shown in Fig. 2 (b).
To solve the first problem, similar to [3], we insert several white frames at the start and the end of source videos for determining the start and the end of the video. For the second problem, we need to insert a flag on the source frames to help distinguish the confused frames and unconfused frames. A naive flag is expanding the source frames with white or black surrounding areas. In this way, we can select the unconfused frames based on the pixel intensity in the surrounding areas. However, this will cause inconsistent brightness of selected frames. Therefore, we propose to insert slash and backslash patterns alternatively surrounding the source frames, as shown in Fig. 3 (a). As shown in Fig. 3 (b), for confused frames, the patterns surrounding the frame will become crossed patterns since displayed frames switch but they are recorded into one frame. Therefore, we can select the unconfused frames based on the patterns, which is formulated as flag recognition step. Specifically, to reduce the influence of noise and moiré artifacts on pattern recognition, we first perform Gaussian filtering on the original captured frames  $(\bar{V}_{\mathrm{raw}})$ , then we improve the contrast of the filtering results via a sigmoid function to make the edge clear. Afterwards, we use Canny edge detection to extract edges on the surrounding area and utilize Hough transform to detect the endpoints of the lines, and calculate the average slope of all lines. If the mean slope value is around 1 (-1), it corresponds to the slash (backslash) pattern. Otherwise, if the slope value is around 0, it corresponds to the cross pattern. In this way, we can remove the confused frames with cross pattern. Then we remove the repetitive frames with the same content by calculating the difference of slope, resulting in the selected raw video  $V_{\mathrm{raw}}$ .
The capturing process, as shown in Fig. 3, can be summarized as follows. Given a set of source frames, we first expand them by alternatively inserting slash and backslash patterns in the surrounding areas. Then, we insert white frames at the beginning and ending of the source frames. For these white frames, we insert a box to help indicate the recording regions in the following frames. The reorganized frames, denoted as  $T_{\mathrm{sRGB}}$  are displayed on the screen with a frame rate of 10 fps. Then, we recapture these frames with mobile phones. Since we need to record the raw frames, we utilize
the MotionCam  $^{2}$  App to help save and export raw frames. The recording frame rate is set to 30 fps. Note that we need to fix the exposure parameters and white balance parameters when shooting, otherwise the color and brightness of recaptured videos will be unstable. The recaptured video frames are denoted as  $\bar{V}_{\mathrm{raw}}$ . Then we select the suitable recaptured frames via deleting the white frames, recognizing the flag pattern, and removing repetitive frames, resulting in  $V_{\mathrm{raw}}$ . In this way, we construct the temporal consistent pairs  $(T_{\mathrm{RGB}}, V_{\mathrm{raw}})$ .
Note that although  $T_{\mathrm{sRGB}}$  and  $V_{\mathrm{raw}}$  are temporal aligned, they have spatial displacements. Therefore, for each video pair, we further utilize spatial alignment strategies to make them pixel-wise aligned. Similar to [34], we first utilize homography transform to coarsely align  $T_{\mathrm{sRGB}}^{t}$  with  $V_{\mathrm{raw}}^{t}$ , where  $t$  is the frame index. Then, we utilize DeepFlow [24] to perform pixel-wise alignment, resulting in  $\hat{T}_{\mathrm{sRGB}}^{t}$ . Finally, we crop the center area of  $\hat{T}_{\mathrm{sRGB}}$ , and  $V_{\mathrm{raw}}$ , generating the video pairs with a resolution of 720p. Note that, we also export the sRGB version of the recaptured videos, denoted as  $V_{\mathrm{sRGB}}$ . Therefore, we also generate the video demoiring pairs in sRGB domain, namely  $\hat{T}_{\mathrm{sRGB}}$  and  $V_{\mathrm{sRGB}}$ .
The displayed source videos are from DAVIS [15], Vimeo $^3$ , high-quality cartoons, and screen recordings, such as scrolling documents, webpages, and GUIs. To ensure the quality of the videos, we select the video whose resolution is ( or larger than) 1080p for recapturing. In this way, our dataset includes natural scenes, documents, GUIs, webpages, etc. We totally generated 300 video pairs and each video contains 60 frames. To ensure the diversity of moiré patterns, we utilize four different combinations of display screens and mobile phones. We change the distances between our camera and the display screen from 10 to 25 centimeters. The view angle changes from 0 to 30 degrees. Please refer to the appendix for more details about our dataset.
Compared with the video demoiring dataset in [3], our dataset contains more diverse contents, and the moiré patterns are also more diverse than that in [3]. In addition, our dataset has no confusion frames while [3] contains confusion frames since its selecting strategy will fail when the frame rate is unstable. We also provide comparisons with synthesized datasets in the appendix. In summary, our dataset contains raw-RGB and RGB-RGB pairs, which can facilitate moiré removal in both raw and sRGB domains.
Figure 3: The pipeline of video preprocessing and recapturing.
4.1 Network Structure Overview
Fig. 4 presents the proposed video and image demoiring network. For videos, in order to utilize the neighboring frame information, the input is three consecutive frames  $\left(V_{\mathrm{raw}}^{t-1}, V_{\mathrm{raw}}^{t}, V_{\mathrm{raw}}^{t+1}\right)$  and the output is the sRGB result of the central frame  $(O_{\mathrm{sRGB}}^t)$ . The whole pipeline includes preprocessing, feature extraction, alignment, temporal fusion, and reconstruction modules. For the preprocessing, we pack the raw Bayer input with a size of  $H \times W$  into four color channels with a size of  $H/2 \times W/2 \times 4$ . Then, we utilize the normal convolution and group convolution (GConv) to generate color-mixed features and color-separated features (i.e., the features for the R, G1, G2, and B channels are generated separately). Following EDVR [20] and VDmoiré [3], we utilize pyramid cascading deformable (PCD) alignment [20] to align the features of  $V_{\mathrm{raw}}^{t-1}$  and  $V_{\mathrm{raw}}^{t+1}$  with those of  $V_{\mathrm{raw}}^t$ . The color-mixed feature and color-separated feature share the same alignment offsets. Then, these features are temporal fused together via a temporal fusion module at multiple scales, which is realized by weighted sum of features from different frames [3]. The fusion weights are predicted from the concatenated temporal features. Afterwards, the fused features go through the reconstruction module to reconstruct the
$^{2}$ https://www)motioncamapp.com/
<sup>3</sup>https://vimeo.com/
demoiréing result at multiple scales. Following [3; 25], the features at different scales are densely connected to make them communicate with each other. Note that, all the operations are applied on two different kinds of features. As shown in the top row of Fig. 4, the first branch is constructed by color-separated features, and the bottom three branches are constructed by color-mixed features. The two kinds of features are fused together at multiple scales with the proposed channel and spatial modulations, which are described in the following. Note that, for image demoiréing, we directly remove the PCD alignment and temporal fusion modules. The remaining modules construct our raw image demoiréing network.
Figure 4: The proposed raw video demoiréing network. The input is three consecutive raw frames  $\left(V_{\mathrm{raw}}^{t-1}, V_{\mathrm{raw}}^{t}, V_{\mathrm{raw}}^{t+1}\right)$  and the output is the multi-scale result of the central frame in sRGB domain, and the first scale result  $O_{\mathrm{sRGB}}^{t_1}$  is the demoiréing result. The downsampling  $(\downarrow)$  is realized by stripped convolution and upsampling  $(\uparrow)$  is realized by bilinear interpolation and convolution. For image demoiréing, the PCD alignment and temporal fusion modules are removed.
4.2 Channel and Spatial Modulations
In the literature, for image demoiréing, we generally utilize the traditional convolutions to convolve with all channels, generating RGB-mixed (or RGBB-mixed for raw inputs) features. This operator can help recover the shared structures of the three channels. However, this also enhances the shared moiré patterns existed in the three channels. For raw images, we observe that the moiré patterns of the four channels in raw domain are different. As shown in Fig. 1, the moiré intensities in the four channels are different and usually exists one channel with fewer moiré patterns. If we only utilize traditional convolution to recover shared structures, the moiré patterns will also spread to all channels. Therefore, besides the traditional convolution, we further introduce the other branch which utilizes Group Convolution (GConv) to process the packed R, G1, G2, and B channels separately. Since the four groups may contribute differently to the demoiréing process, we further introduce learnable parameters to modulate the four groups (i.e., channel-wise multiplication), as shown in Fig. 5. Then, the color-mixed features and color-separated features are fused together via element-wise addition. In this way, we can combine the advantages of the two features.
Afterwards, we further introduce spatial modulation. The moiré patterns usually have a large spatial period. The receptive field of small convolution filter is limited. Therefore, we propose to utilize the feature with large receptive field to modulate the feature with small receptive field. Inspired by [7], we directly utilize the convolution based modulation to realize this process. As shown in Fig. 5, we first utilize the Linear operation (realized by  $1 \times 1$  conv) to transform the fused features. Then, the top branch goes through depth-wise convolution (DConv) with a large kernel  $(11 \times 11)$ , and the generated feature is used to modulate the feature in the bottom branch. Afterwards, the modulated feature further goes through a Linear layer to introduce cross-channel communication which is absent for DConv. After channel and spatial modulation (CSMod), the color-separated feature and color-mixed feature are well fused and enhanced, which are then fed into the following Conv blocks for demoiring reconstruction.
4.3 Loss Functions
We observe that two-stage training is better for our network. Therefore, we utilize a two-stage training strategy. At the first stage, the baseline network (namely removing the color-separated feature and CSMod from Fig. 4) is trained with  $\ell 1$  loss and perceptual loss. It is formulated as (for brevity, we
Figure 5: Channel and spatial modulation (CSMod) block.
omit the subscript sRGB for  $O^t$  and  $\hat{T}^t$ )
$$
\mathcal {L} _ {s 1} = \sum_ {i = 1} ^ {i = 3} \lambda_ {1} \left\| O ^ {t _ {i}} - \hat {T} ^ {t _ {i}} \right\| _ {1} + \lambda_ {2} \left\| \psi_ {j} \left(O ^ {t _ {i}}\right) - \psi_ {j} \left(\hat {T} ^ {t _ {i}}\right) \right\| _ {1}, \tag {1}
$$