SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\hat{T}^{t_i}$  is the ground truth and  $i$  represents the scale index.  $\psi_{j}$  represents the VGG features extracted at the  $j^{th}$  layer, and we utilize the conv1_2 layer in our implementation.  $\lambda_{1}$  and  $\lambda_{2}$  are the weighting parameters. This stage can remove the moiré patterns to some extent but fail for hard samples. For example, if the moiré patterns are strong for one color channel, it is difficult to be removed with only the color-mixed features. Therefore, in the second stage, we introduce the color-separated feature and CSMo to the baseline, and train the whole network. Since we need to recover correct color tones in sRGB domain, similar to [34], we further introduce color loss. Namely, the second stage loss function is
$$
\mathcal {L} _ {s 2} = \sum_ {i = 1} ^ {i = 3} \lambda_ {3} \| O ^ {t _ {i}} - \hat {T} ^ {t _ {i}} \| _ {1} + \lambda_ {4} \| \psi_ {j} (O ^ {t _ {i}}) - \psi_ {j} (\hat {T} ^ {t _ {i}}) \| _ {1} + \lambda_ {5} \left(1 - \frac {1}{N} \sum_ {k} \frac {O _ {k} ^ {t _ {i}} \hat {T} _ {k} ^ {t _ {i}}}{\| O _ {k} ^ {t _ {i}} \| \| T _ {k} ^ {t _ {i}} \|}\right), \tag {2}
$$
where  $k$  is the pixel index, and  $N$  is the pixel number.  $\lambda_3$ ,  $\lambda_4$ , and  $\lambda_5$  are the weighting parameters.
5.1 Implementation Details and Datasets
For video-demorieing, the weighting parameters  $\lambda_{1} - \lambda_{5}$  in Eq. 1 and 2 are set to 0.5, 1, 5, 1, 1, respectively. The first stage training starts with a learning rate of 2e-4, which decreases to 5e-5 at the 37th epoch. The baseline reaches convergence after 40 epochs. In the second stage, the initial learning rate is 2e-4, which decreases to 1e-4 and 2.5e-5 at the 10th and 30th epochs. All the network parameters are optimized by Adam. The batch size is set to 14 and the patch size is set to 256. Image demorieing shares similar training strategies. The proposed method is implemented in PyTorch and trained with two NVIDIA 3090 GPUs.
For image demoiring, we utilize the dataset constructed by [34], which contains raw image inputs. For video demoiring, we utilize the dataset constructed in Sec. 3, which contains 300 video clips. We randomly select 50 video clips to serve as the testing set.
Table 1: Comparison with state-of-the-art image demoiring methods in terms of average PSNR, SSIM, LPIPS, and computing complexity. The best results are highlighted in bold.
Method	PSNR↑	SSIM↑	LPIPS↓	Parameters (M)	GFLOPs
FHDe2Net	23.630	0.8720	0.1540	13.59	353.37
RDNet	26.160	0.9210	0.0910	6.045	10.06
MBCNN	26.667	0.9272	0.0972	2.722	9.74
MBCNN*	27.187	0.9332	0.0888	2.717	9.67
UHDM	26.513	0.9246	0.0911	5.934	17.56
UHDM*	27.078	0.9302	0.0812	5.925	17.40
Ours	27.261	0.9346	0.0748	5.327	21.65
Table 2: Comparison with state-of-the-art image and video demoiring methods for video demorieding in terms of average PSNR, SSIM, LPIPS, and computing complexity. The best result is highlighted in bold and the second best is underlined.
Method	PSNR↑	SSIM↑	LPIPS↓	Parameters (M)	GFLOPs
Image	MBCNN	25.892	0.8939	0.1508	2.722	9.74
MBCNN*	26.082	0.8984	0.1402	2.717	9.67
UHDM	26.823	0.9078	0.1177	5.934	17.56
UHDM*	25.642	0.8792	0.1232	5.925	17.40
Video	MFrame-RDNet	26.970	0.9051	0.1176	6.640	34.31
MFrame-UNet	26.185	0.8917	0.1368	32.286	39.01
VDMoiré	27.277	0.9071	0.1044	5.836	171.93
VDMoiré*	27.747	0.9116	0.0995	5.838	43.02
EDVR	26.785	0.8993	0.1172	2.857	990.74
EDVR*	27.285	0.9064	0.1053	3.005	257.50
VRT-	24.903	0.8728	0.1387	2.485	353.37
VRT-*	27.113	0.9034	0.1091	2.733	101.60
Ours	28.706	0.9201	0.0904	6.585	70.61
5.2 Comparison for Image Demoiring
For image demoiréing, we compare with the raw domain demoiréing method [34] and the state-of-the-art sRGB domain image demoiréing methods, i.e.,  $\mathrm{FHDe}^2\mathrm{Net}$  [6], MBCNN [41], and UHDM [28]. All these methods are retrained on the raw image dmoiréing dataset [34]. For the most competitive methods MBCNN and UHDM, we further retrain them with raw inputs by revising (changing the input channel number and inserting upsampling layers) the network to make it adapt to raw inputs. The revised versions are denoted as MBCNN* and UHDM*. Since PSNR is heavily affected by the reconstructed color tones, SSIM and LPIPS [38] are good complementary measurements for comprehensive evaluation. As shown in Table 1, our method achieves the best results on all three measurements. Fig. 6 presents the visual comparison results for the top six methods. It can be observed that all the compared methods fail when deal with the regular and strong moiré patterns. In contrast, our method can remove them clearly.
5.3 Comparison for Video Demoiring
To our knowledge, there is only one work, i.e., VDMoiré [3], dealing with video demoiréing. Therefore, we further include other video restoration methods and state-of-the-art image demoiréing methods for comparison. Specifically, we introduce the benchmark CNN based and transformer based video super-resolution networks, i.e., EDVR [20] and VRT [8], for comparison. Since VRT is very large, we reduce its block and channel numbers for a fair comparison, and the revised version is denoted as  $\mathrm{VRT^{-}}$ . The original loss functions of EDVR and VRT are changed to our loss functions for better demoiréing performance. For image demoiréing, we introduce MBCNN [41] and UHDM [28] for comparison. Since all the compared methods are designed for sRGB inputs, we first retrain them on our RawVDemoiré dataset with sRGB domain pairs. Then, we retrain them with our raw-sRGB pairs. We utilize superscript * to denote the raw input versions. In addition, we transform two raw image processing methods (RDNet [34] and UNet [16]) for video demoiréing by incorporating the PCD module and fusion module. The revised versions are denoted as MFrame-RDNet and MFrame-UNet. The inputs of the two modules are three consecutive raw frames.
Figure 6: Comparison with state-of-the-art image demoiring methods.
Table 2 presents the quantitative comparison results. All the raw domain processing methods (except UHDM*) are better than their corresponding sRGB domain versions. All the image based demoiréing methods cannot achieve satisfied results for video demoiréing since they cannot utilize temporal correlations. For EDVR, since it does not reconstruct multiscale results, it cannot remove large moiré patterns. VRT $^{-}$  also cannot generate satisfied results since the shift-window matching mechanism of VRT is not beneficial for large moiré removal. In addition, the revised multi-frame based raw image processing methods MFrame RDNet and MFrame UNet still cannot generate satisfied results. In contrast, our method generates the best demoiréing results, outperforming the second best method by nearly 1 dB. The main reason is that our CSMoD strategy can well utilize the complementary information of RGGB channels and can remove strong moiré patterns.
We also give the computing complexity comparison in terms of parameter numbers and GFLOPs in Table 2. It can be observed that the image demoiring methods consume small computing costs since they process the video frame-by-frame, which also leads to worse demoiring results. Our GFLOPs are smaller than the popular video restoration frameworks, such as EDVR and VRT, which indicates our method is an efficient video demoiring method.
Fig. 7 presents the visual comparison results for one video frame. It can be observed that our method can remove the moiré patterns clearly while the compared methods all fail for this hard sample. More results are provided in our appendix.
Figure 7: Comparison for video demoiring.
Table 3: Ablation study results by adding the proposed modules one by one. The best results are highlighted in bold.
Variant	PSNR↑	SSIM↑	LPIPS↓
Baseline	27.963	0.9116	0.0914
Baseline+SMod	28.439	0.9175	0.0939
Baseline+SMod+	28.498	0.9182	0.0957
Baseline+CSMod	28.706	0.9201	0.0904
Table 4: Ablations on loss functions and the multi-scale constraint. The best results are highlighted in bold.
Variant	PSNR↑	SSIM↑	LPIPS↓
w/o VGG Loss	28.069	0.9166	0.1160
w/o Color Loss	28.642	0.9169	0.0931
w/o Multi-scale Loss	27.455	0.9124	0.1041
Complete	28.706	0.9201	0.0904
Ablation on SMod and CSMod. We perform ablation study by adding the proposed modules one by one. The first variant is our baseline network. The second is introducing the proposed spatial modulation into the baseline network. Namely we only keep the color-mixed features in the channel modulation module in Fig. 5. The third is introducing SMod and replacing the color-separated feature in Fig. 5 with another color-mixed feature (denoted as  $\mathrm{SMod}^+$ ). The fourth is our complete version, namely Baseline+CSMod. As shown in Table 3, the SMod improves the demoiring performance by  $0.48\mathrm{dB}$ . One reason is that the SMod can take advantage of large receptive field and the other reason is that we utilize the second stage training loss to update SMod, which is beneficial for color recovery. Compared with SMod,  $\mathrm{SMod}^+$  only achieves little improvement, which demonstrates that the color-separated features are essential. Compared with SMod, the proposed CSMod can further improve  $0.26\mathrm{dB}$ .
Ablation on Loss Functions. We further conducted ablation study regarding perceptual loss (VGG Loss), color loss (Color Loss), and multi-scale constraints (Multi-scale Loss), and the results are summarized in Table 4. When the corresponding loss is removed, there is a noticeable decrease in all
evaluation metrics. Perceptual loss places constraints on deep-level features of images, which helps recover structural information, such as shapes and textures. Color loss, on the other hand, focuses more on the global color of images. Recaptured screen images often suffer from weakened attributes like brightness, contrast, color saturation, and details. The constraint provided by L1 loss alone might not be sufficient. Incorporating perceptual loss and color loss can contribute to better image reconstruction quality. Given the multi-scale characteristics of moiré patterns, imposing multi-scale constraints is beneficial for moiré removal and image reconstruction. Note that, the multi-scale constraints are widely utilized in benchmark demoiring models, such as MBCNN, UHDM, and VDMoiré. In addition, the perceptual loss is also included in all of our compared methods (except MBCNN, which utilizes a more advanced loss function).
5.5 Generalization
The image demoiring experiment is conducted on the dataset provided by [34], whose test set contains two device combinations that are not included in the training set. Therefore, we utilize these test images (about 1/4 of the whole test set) to evaluate the generalization ability of different models. As shown in Table 5, our method still achieves the best results in terms of SSIM and LPIPS, and the PSNR value is only slightly inferior to UHDM*. Note that, for our method, if the test camera-screen combination exists in the training set, the color tone of the test images can be well recovered. Otherwise, the recovered image may have color cast. The main reason is that our network also performs the ISP process, and the ISP process is indeed different for different cameras. Fortunately, the moiré patterns can still be removed. Therefore, our LPIPS value is still small. In summary, our method has the best generalization ability in terms of moiré removal.
Table 5: Comparison of the generalization ability of different models by evaluating with the screen-camera combinations that are not included in the training set. The best results are highlighted in bold and the dataset is the image demoriéing dataset [34].
Method	PSNR↑	SSIM↑	LPIPS↓
FHDe2Net	22.840	0.8968	0.1530
RDNet	24.798	0.9280	0.0846
MBCNN	24.726	0.9253	0.1017
MBCNN*	24.659	0.9314	0.0925
UHDM	24.679	0.9239	0.0955
UHDM*	25.769	0.9310	0.0759
Ours	25.719	0.9333	0.0727
In this paper, we propose a novel image and video demoiréing network tailored for raw inputs. We propose an efficient channel modulation block to take advantages of color-separated features and color-mixed features. The spatial modulation module helps our model utilize large receptive field information with fewer computation costs. In addition, we construct a RawVDemoiré dataset to facilitate research in this area. Experimental results demonstrate that our method achieves the best results for both image and video demoiréing.
Our work is the first exploring raw video demoiréing and we construct the first RawVDemoiré dataset. To solve the temporal confusion problem during video recapturing, we propose an efficient temporal alignment method, which can be extended to other screen recapturing scenarios. Previous works usually omit the differences between raw and sRGB inputs. Our work is an exploratory work that takes advantage of color-separated features in raw domain and we expect more works along this direction. However, we would like to point out that the demoiréing results may be used for face-spoofing since the moiré detection method cannot identify our result. But on the other hand, our method can imitate the attacks for face recognition, which can help train robust antispoofing methods.