SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Quantitative Results. In DVSOD, one important expectation compared to the RGB-based model is whether properly utilizing RGB-D and temporal features improves the per-frame segmentation accuracy. To verify it, we first conduct three comparative experiments (last three columns of Table 2) to evaluate the influence of each data type. The basic model, CPD [73], which is trained with just RGB images, achieves an overall Mean Absolute Error (MAE) score of  $13.2\%$ . When we incorporate a depth component  $(\mathrm{CPD} + \mathrm{Dep})$ , the error significantly decreases by  $1.1\%$ . As we employ multimodal RGB-D data and temporal video data jointly  $(\mathrm{CPD} + \mathrm{DepVid})$ , the performance further improves to  $11.3\%$ . These results consistently highlight the advantages of using 3D layout data and temporal contexts to locate salient objects. Considering that DVSOD is relatively new and its development is still at an early stage, we additionally present the segmentation results of related SOD/RGB-D SOD/VSOD models in Table 2, to provide observational reference for our readers.
Qualitative Results. Fig. 5 visualizes the saliency predictions from different saliency models on the DViSal dataset. We find that the model trained with both RGB-D data and video data generates saliency maps that closely match the ground truths. For example, the DVSOD baseline effectively identifies entire objects even in challenging low-light situations as displayed in Fig. 5 where other methods struggle. We attribute this success to the combined benefits of multimodal and temporal contexts. In addition, our DViSal dataset's diverse scenarios in Figs. 1-2&5 demonstrate its versatility to provide a sufficiently realistic benchmark in this field.
Table 2: Benchmarking results of related SOD approaches tested on the newly-proposed DViSal dataset. We report the numerical evaluation for each test subset and provide an overall measurement.  $\uparrow \& \downarrow$  denote larger and smaller is better, respectively. The notations  $\dagger$  and  $\ddagger$  refer to RGB-D SOD and VSOD models, respectively. The best performing model is highlighted in bold.
*	Model	PoolNet [43]	BASNet [65]	F3Net [72]	DMRA [62]†	CoNet [28]†	BBSNet [16]†	RD3D [7]†	SPNet [95]†	PCSA [19]‡	UGPL [63]‡	CPD [73]	CPD +Dep†	CPD +Dep Vid†‡
DET	Eξ↑	.762	.761	.766	.768	.775	.784	.786	.788	.771	.781	.761	.781	.782
Sα↑	.666	.672	.688	.678	.696	.692	.695	.686	.680	.688	.677	.683	.705
Fβ↑	.520	.528	.543	.532	.549	.563	.562	.565	.535	.552	.532	.554	.568
M↓	.143	.142	.146	.135	.139	.134	.127	.126	.134	.136	.142	.139	.129
Scene	Eξ↑	.769	.826	.905	.838	.830	.906	.852	.888	.837	.868	.804	.880	.881
Sα↑	.646	.682	.813	.738	.772	.802	.722	.728	.738	.768	.719	.786	.825
Fβ↑	.441	.514	.751	.571	.652	.759	.575	.749	.569	.721	.535	.733	.722
M↓	.093	.075	.043	.065	.062	.043	.065	.049	.067	.050	.078	.048	.043
People	Eξ↑	.472	.467	.517	.474	.488	.528	.515	.574	.478	.581	.409	.572	.584
Sα↑	.492	.476	.510	.529	.502	.553	.551	.532	.541	.550	.463	.511	.558
Fβ↑	.162	.114	.160	.192	.138	.178	.198	.192	.192	.224	.106	.169	.207
M↓	.153	.146	.119	.139	.128	.086	.103	.092	.128	.078	.254	.098	.085
Track3D	Eξ↑	.657	.600	.780	.634	.658	.839	.655	.860	.638	.838	.612	.831	.822
Sα↑	.590	.589	.779	.629	.655	.724	.633	.667	.629	.693	.600	.726	.751
Fβ↑	.301	.290	.567	.343	.423	.664	.369	.682	.342	.657	.322	.621	.609
M↓	.127	.145	.087	.118	.101	.080	.117	.083	.116	.080	.134	.080	.073
Tracklam	Eξ↑	.815	.798	.913	.828	.896	.923	.809	.889	.833	.924	.787	.915	.925
Sα↑	.701	.682	.769	.717	.761	.793	.697	.739	.717	.785	.702	.771	.804
Fβ↑	.622	.595	.779	.652	.751	.809	.628	.759	.659	.803	.608	.792	.795
M↓	.122	.133	.076	.114	.082	.068	.127	.084	.112	.063	.133	.069	.063
PTB	Eξ↑	.908	.880	.909	.893	.913	.924	.925	.911	.894	.919	.888	.906	.927
Sα↑	.793	.791	.826	.810	.824	.834	.811	.793	.808	.832	.816	.819	.842
Fβ↑	.826	.710	.795	.753	.808	.821	.843	.835	.754	.837	.745	.816	.833
M↓	.062	.067	.057	.060	.056	.054	.053	.060	.060	.052	.058	.052	.049
Overall	Eξ↑	.774	.774	.795	.783	.794	.810	.799	.810	.786	.806	.772	.805	.807
Sα↑	.673	.679	.711	.692	.713	.715	.703	.698	.694	.709	.689	.705	.729
Fβ↑	.537	.540	.589	.555	.585	.609	.582	.608	.557	.598	.548	.599	.610
M↓	.133	.132	.128	.125	.124	.118	.118	.113	.123	.119	.132	.121	.113
Diagnostic Analysis. Here we discuss the impact of memory size  $\mathcal{M}$  and sample rate  $S$  for memory frame selection. As shown in Table 3, adding memory frames consistently improves  $F_{\beta}$  scores compared to the single-frame model (i.e.,  $\mathcal{M} = 0$ ). When using more memory frames (i.e.,  $\mathcal{M} = 3$ ), we see a clear performance increase (i.e.,  $59.9\% \rightarrow 61.0\%$ ). Increasing the  $\mathcal{M}$  further beyond 3 gives marginal returns in performance. Hence, we set  $\mathcal{M} = 3$  as a balance between accuracy and memory usage. We then tested different sample rates while fixing memory size  $\mathcal{M} = 3$ . It is discovered in Table 4 that the best performance is achieved with a moderate sample rate  $S = 3$ . Thus, we set both  $\mathcal{M}$  and  $S$  to 3, effectively using past video frames without retaining too much outdated information.
Table 3: Ablation on the impact of memory size using F-measure and MAE.
*	M=0	M=1	M=2	M=3	M=4
Fβ↑	0.599	0.603	0.607	0.610	0.608
M↓	0.121	0.118	0.114	0.113	0.113
Table 4: Ablation on the impact of sample rate using F-measure and MAE.
*	S=1	S=2	S=3	S=4
Fβ↑	0.605	0.608	0.610	0.605
M↓	0.117	0.114	0.113	0.114
5 Discussion and Outlook
In this section, we discuss several potential research problems on the emerging  $DVSOD$  task, and provide some feasible solutions for reference. Hopefully this could encourage more inspirations and contributions to this community. They are summarized as follows:
Figure 5: Visual results of different SOD models. The  $\mathrm{CPD}^{+\mathrm{DepVid}}$ , stands for the DVSOD baseline in Sec. 4.1 that integrates both multimodal RGB-D cues and temporal contexts.
depict DVSOD model performance due to the complex scenes within the DViSal benchmark, such as low light or cluttered backgrounds. Thus, how to design appropriate metrics specifically tailored for DVSOD remains an open challenge.
(6) Instance-level Extension: As revealed in [18], instance-level extension is capable of recognizing individual instances of salient objects. It is more challenging as it requires detailed parsing within the detected salient regions, and holds significant importance for practical applications. Hence exploring the research on RGB-D video salient instance detection emerges as a promising future direction.
(7) Weakly-supervised Learning: The acquisition of precise per-pixel labels is laborious and time-consuming. As an alternative, training DVSOD models with weak supervisions [80] (e.g., bounding box, scribble) is an appealing research topic, which can avoid heavy annotation costs.
In this work, we present the DViSal dataset to spur research in RGB-D video salient object detection (DVSOD), a field yet to be well explored. This dataset, composed of 237 diverse RGB-D videos and comprehensive annotations, will significantly expand research avenues. Empirical benchmarking experiments demonstrate the effectiveness of multimodal video input in enhancing salient object detection. We highlight some promising future research directions and make the dataset and results publicly available to accelerate progress in this field.
Acknowledgements. This work was done jointly by UA and SRA. This research was supported by the CFI-JELF, Mitacs, University of Alberta Start-up grant, NSERC Discovery (RGPIN-2019-04575) grants.
[1] Radhakrishna Achanta, Sheila Hemami, Francisco Estrada, and Sabine Susstrunk. Frequency-tuned salient region detection. In CVPR, pages 1597-1604. IEEE, 2009.
[2] Sharon Alpert, Meirav Galun, Ronen Basri, and Achi Brandt. Image segmentation by probabilistic bottom-up aggregation and cue integration. In CVPR, pages 1-8, 2007.
[3] Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: A benchmark. IEEE Transactions on Image Processing, 24(12):5706-5722, 2015.
[4] Ali Borji, Dicky N Sihite, and Laurent Itti. What stands out in a scene? a study of human explicit saliency judgment. Vision Research, 91:62-77, 2013.
[5] Tim Caselitz, Michael Krawez, Jugesh Sundram, Mark Van Loock, and Wolfram Burgard. Camera tracking in lighting adaptable maps of indoor environments. In ICRA, pages 3334-3340, 2020.
[6] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, pages 801-818, 2018.
[7] Qian Chen, Ze Liu, Yi Zhang, Keren Fu, Qijun Zhao, and Hongwei Du. RGB-d salient object detection via 3d convolutional neural networks. In AAAI, pages 1063-1071, 2021.
[8] Ming-Ming Cheng, Niloy J Mitra, Xiaolei Huang, Philip HS Torr, and Shi-Min Hu. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):569-582, 2014.
[9] Yupeng Cheng, Huazhu Fu, Xingxing Wei, Jiangjian Xiao, and Xiaochun Cao. Depth enhanced saliency detection method. In Proceedings of International Conference on Internet Multimedia Computing and Service, pages 23-27, 2014.
[10] François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, pages 1251-1258, 2017.
[11] Maximilian Denninger, Dominik Winkelbauer, Martin Sundermeyer, Wout Boerdijk, Markus Wendelin Knauer, Klaus H Strobl, Matthias Hunt, and Rudolph Triebel. Blenderproc2: A procedural pipeline for photorealistic rendering. Journal of Open Source Software, 8(82):4901, 2023.
[12] Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Structure-measure: A new way to evaluate foreground maps. In ICCV, pages 4558-4567, 2017.
[13] Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, and Ali Borji. Enhanced-alignment measure for binary foreground map evaluation. In IJCAI, pages 698-704, 2018.