SlowGuess/NeurIPS_2023_OCR huggingface dataset:
by masks will be separately input into the recovery framework. Then, we propose to use several transformer encoder layers [61] to fuse and re-encode these two features to achieve feature completion. By attention-based decoding and channel fusion, an intermediate feature is generated. Consequently, with skip connection and output projection, the representative capability of the resulting feature can be further enhanced by fusing multi-scale and multi-level information. We then follow the approach of flow-guided video inpainting to extract and complete optical flows from neighboring frames to serve as guidance for feature alignment and propagation. Afterward, a content generation module based on temporal focal transformer and soft splitting will combine the enhanced, aligned, and propagated features of local neighboring frames with the reference features of non-local frames' corruption-free regions  $\{R^t\in \mathcal{R}^{3\times h\times w}|t = 1,2,\dots,T_{nl}\}$  to generate content and finally reconstruct a result frame sequence  $\{\hat{Y}^t\in \mathcal{R}^{3\times h\times w}|t = 1,2,\dots,T_l\}$  through a decoder  $(D)$  module. More detailed descriptions of the methodology can be found in the Supplementary Material.
The proposed BSCVR and state-of-the-art (SOTA) video inpainting methods are performed on the constructed BSCV dataset. We conduct comprehensive quantitative and qualitative evaluations to demonstrate the effectiveness of our dataset and method. The flexibility of our corruption model and the robustness of our BSCVR framework are validated on multiple branches of BSCV.
Experimental setting. We adopt the corruption parameter of (1/16, 0.4, 4096), and its corresponding statistics have been illustrated in Fig. 3. This setting has moderate difficulties with adequate corruption types, which is suitable for video inpainting methods. The corrupted region is usually recoverable yet challenging. SOTA video inpainting methods are compared with our method to recover corruption-free videos. STTN [28] and FuseFormer [16] downsample videos to 240P due to the limitation of computational complexity. E2FGVI-HQ is an upgraded version of E2FGVI [15], stating that it could take arbitrary resolution and generate results with the original input resolution. They could be viewed as the main competitor of our method. Noted that previous works set the mask with "random shape and location" to augment inpainting data, and those pre-trained models are trained with 500K iterations. In contrast, training those methods on our dataset requires only 250K iterations to converge. The detailed implementation of our method can refer to the Supplementary Material.
Test res.	Method	Accuracy	Efficiency
YouTube-VOS (720P) Subset	DAVIS (480P) Subset	Runtime s/frame
PSNR↑	SSIM↑	LPIPS↓	VFID↓	PSNR↑	SSIM↑	LPIPS↓	VFID↓
240P	Input	18.8749	0.8160	0.1527	0.2015	18.4562	0.7921	0.1541	0.4189	-
STTN [28]	29.3840	0.9174	0.0465	0.0566	26.2172	0.8600	0.0638	0.1589	0.120
STTN [28]*	29.9172	0.9303	0.0394	0.0544	26.5453	0.8722	0.0575	0.1534	0.120
FuseFormer [16]	28.8012	0.9047	0.0549	0.0641	26.2547	0.8618	0.0659	0.1645	0.200
FuseFormer [16]*	29.8108	0.9328	0.0381	0.0526	26.7367	0.8834	0.0531	0.1477	0.200
E2FGVI-HQ [15]	29.6866	0.9228	0.0469	0.0555	26.7850	0.8765	0.0600	0.1513	0.160
E2FGVI-HQ [15]*	31.0030	0.9473	0.0341	0.0479	27.6551	0.9018	0.0491	0.1387	0.160
BSCVR-S (Ours)*	31.8345	0.9584	0.0262	0.0427	28.4211	0.9180	0.0381	0.1196	0.172
BSCVR-P (Ours)*	31.9534	0.9598	0.0258	0.0426	28.5430	0.9199	0.0375	0.1165	0.178
Ori-ginal	Input	19.1490	0.8244	0.1415	0.0575	18.4384	0.7979	0.1490	0.1999	-
E2FGVI-HQ [15]	28.5039	0.8783	0.0453	0.0126	25.7803	0.8236	0.0504	0.0468	0.192 / 0.176
E2FGVI-HQ [15]*	29.5666	0.9023	0.3955	0.0161	26.6723	0.8611	0.0530	0.0577	0.192 / 0.176
BSCVR-S (Ours)*	30.2235	0.9185	0.0335	0.0143	27.2770	0.8809	0.0427	0.0500	0.250 / 0.203
BSCVR-P (Ours)*	29.9943	0.9144	0.0343	0.0104	26.3564	0.8511	0.0416	0.0406	0.261/ 0.213
Table 2: Quantitative results of SOTA pre-trained video inpainting methods, their corresponding models trained on our dataset (denoted by *), and our method. In our method, BSCVR-S means that the feature completion module considers the input feature as a sequence like traditional Transformer [61], and BSCVR-P indicates that the module considers the input as patches referring to SwinIR [31]. The comparison is conducted under the 240P setting due to the model capability of previous works. For the methods which are able to handle arbitrary-resolution video, we calculate metrics based on the original frame sequence, and we measured and demonstrate the runtime of the model under 720P (former) / 480P (latter) input, respectively.
Evaluation metrics. Regarding the quantitative evaluation, we measure the performance of inpainting algorithms based on two aspects: inpainting quality reconstruction and realism. Among them, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [62], and Learned Perceptual Image Patch Similarity (LPIPS) [63] using pre-trained AlexNet backbone [64] are mainly used to measure reconstruction performance. Video Fréchet Inception Distance (VFID) [65] is mainly used to measure performance in terms of realism. It corresponds to the sets of all recovered videos and all reference videos. The features are extracted from a pre-trained I3D backbone [66].
Quantitative evaluation. The quantitative results on the YouTube-VOS and DAVIS subsets are shown in Table 2. Due to the model capability of some previous works, we first follow the existing experimental setting of video inpainting to downscale the original video in our dataset to 240P and conducted model training and metric calculation. It can be observed that our methods achieve better results in all metrics. However, compromising on video resolution is not reasonable for the video recovery problem. Thus, we particularly compared our method with the SOTA method E2FGVI-HQ [15] which is currently the only method can handle
Param. (P, L, S)	Methods	Accuracy
DAVIS Subset
PSNR†	SSIM†	LPIPS↓	VFID↓
(1/16, 0.4, 4096)	Input	18.4384	0.7979	0.1490	0.1999
E2FGVI-HQ [15]*	26.3734	0.8415	0.0466	0.0444
BSCVR-S (Ours)	27.2770	0.8809	0.0427	0.0500
BSCVR-P (Ours)	26.3564	0.8511	0.0416	0.0406
(1/16, 0.4, 2048)	Input	18.2283	0.7798	0.1536	0.1920
E2FGVI-HQ [15]*	26.0251	0.8423	0.4777	0.0440
BSCVR-S (Ours)	26.2437	0.8554	0.0416	0.0401
BSCVR-P (Ours)	26.1057	0.8525	0.0422	0.0407
(1/16, 0.2, 4096)	Input	18.0789	0.7649	0.1569	0.2067
E2FGVI-HQ [15]*	24.7468	0.7746	0.0656	0.0546
BSCVR-S (Ours)	24.9832	0.7980	0.0570	0.0478
BSCVR-P (Ours)	24.8303	0.7940	0.0579	0.0484
(2/16, 0.4, 4096)	Input	17.9418	0.7616	0.1592	0.1963
E2FGVI-HQ [15]*	24.3774	0.7934	0.0623	0.0767
BSCVR-S (Ours)	24.5066	0.8077	0.0553	0.0698
BSCVR-P (Ours)	24.3808	0.8037	0.0561	0.0705
(1/16, 0.4, 8192)	Input	18.6665	0.8170	0.1450	0.1849
E2FGVI-HQ [15]*	26.0722	0.8371	0.0486	0.0455
BSCVR-S (Ours)	26.2708	0.8518	0.0423	0.0417
BSCVR-P (Ours)	26.1231	0.8487	0.0430	0.0418
(1/16, 0.8, 4096)	Input	19.0062	0.8419	0.1389	0.1874
E2FGVI-HQ [15]*	32.8311	0.9506	0.0162	0.0264
BSCVR-S (Ours)	32.7959	0.9514	0.0147	0.0247
BSCVR-P (Ours)	32.7204	0.9509	0.0149	0.0252
(4/16, 0.4, 4096)	Input	17.8542	0.7587	0.1610	0.1973
E2FGVI-HQ [15]*	22.7912	0.7480	0.0738	0.1192
BSCVR-S (Ours)	22.7094	0.7570	0.0679	0.1079
BSCVR-P (Ours)	22.5480	0.7527	0.0686	0.1079
Table 3: Performance comparison with E2FGVI-HQ on different branches with varied corruption parameter combinations.
(a) Qualitative results comparing our method with STTN [28], FuseFormer [16], and E2FGVI-HQ [15] under unified 240P resolution.
(b) Qualitative results comparing our method with E2FGVI-HQ [15] trained on our dataset under original video resolution.
Figure 5: Qualitative comparison of our method and SOTA video inpainting methods on low (a) and high (b) resolutions. The involved corruption types include (1) blocking artifacts, (2) color artifacts, (3) duplication artifacts, (4) misalignment, (5) texture loss, (6) trailing Artifacts and their combinations.
the original resolution scenario. For the metrics of PSNR, SSIM, LPIPS, and VFID, our method can achieve significant improvement, which makes the bitstream-corrupted video recoverable, e.g.,  $>30\mathrm{dB}$  PSNR for YouTube-VOS. Our method could comprehensively refine the content in corrupted regions to guide plausible content generation and keep low computational complexity by applying efficient model architecture referring to E2FGVI-HQ [15]. For the results on different corruption parameters and more comparison with latest non-end-to-end methods, we provided more experiments results and discussion in the Supplementary Material.
Qualitative Evaluation. For qualitative evaluation, we choose STTN [28], FuseFormer [16], and E2FGVI-HQ [15] trained on our dataset as comparison methods. The evaluation is conducted under 240P and original resolutions. Some representative corruption patterns and their recovery results are visualized in Fig. 5a. The comparison methods are difficult to generate plausible content to recover the corrupted region, while our proposed method can generate clearer details with more informative textures and structures. Our method and dataset limits the tendency of object removal when the mask is relatively large, especially under the original resolution and the compared results of our method with the SOTA method E2FGVI-HQ [15] are shown in Fig. 5b. These results demonstrate
Figure 6: Corruption ratio distribution of the YouTube-VOS&DAVIS dataset branches for experiments under different corruption parameter combinations.
the limitation of current methods on the proposed dataset and the advantage of our high quality data and method. More visualized results can be found in the Supplementary Material.
Flexibility in Dataset Construction. We use the three-parameter corruption model  $(P, L, S)$  in our dataset construction. We validate the model by generating more branches of dataset with seven additional parameter settings: (1/16, 0.4, 2048), (1/16, 0.4, 8192), (1/16, 0.4, 4096), (1/16, 0.2, 4096), (1/16, 0.8, 4096), (2/16, 0.4, 4096) and (4/16, 0.4, 4096). These settings represent varying corruption probabilities, where  $P = m / l$  implies that the corruption happens in random  $m$  frames out of  $l$  frames of a GOP.
We analyze the corruption distribution by calculating the ratio of the corrupted region to the frame resolution, as shown in Fig. 6. It indicates that parameter  $P$  has the most significant impact on corruption, leading to a higher number of corrupted frames and more severe damage. For these dataset branches, we compare our BSCVR and E2FGVI-HQ under the original resolution on the DAVIS 480P subset. The results are listed in Tab. 3. It shows that the proposed BSCVR consistently outperforms E2FGVI-HQ, validating the robustness of our BSCVR on multiple dataset branches.
Aiming at the challenging problem of bitstream-corrupted video recovery in the real world, we construct the first large-scale benchmark, BSCV. The BSCV provides a bitstream corruption model, a realistic decoded video dataset, and a video recovery framework, BSCVR. The bitstream corruption model enables to flexibly generate dataset branches by specifying parameter combinations. The dataset contains 28,000 realistic video clips decoded from corrupted bitstreams with unpredictable error patterns and corruption levels. The BSCVR offers a effective framework for high-quality video recovery. Extensive experiments demonstrate that the proposed BSCVR outperforms SOTA video inpainting methods quantitatively and qualitatively. The flexibility of dataset construction and the robustness of our BSCVR framework are also validated on various dataset branches. The benchmark dataset is expected to benefit video recovery in multimedia forensics and video communication application includes live streaming and online conference, etc. The future work will concentrate on designing more reasonable bitstream corruption models, engaging more dataset sources, and creating more effective recovery frameworks.
This research is supported by the National Research Foundation, Singapore, and Cyber Security Agency of Singapore under its National Cybersecurity Research & Development Programme (Cyber-Hardware Forensic & Assurance Evaluation R&D Programme <NRF2018NCRNCR009-0001>). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the view of National Research Foundation, Singapore and Cyber Security Agency of Singapore.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[62] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
[63] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[64] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[65] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5792–5801, 2019.
[66] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.