(1) Accuracy: The research of DVSOD is still in its initial stage and DVSOD research has room for improvement, particularly regarding the accuracy of the DVSOD models. There's potential to enhance the accuracy of DVSOD models by incorporating ideas from the established fields of RGB, RGB-D, and Video SODs. For instance, we could integrate the multi-scale learning techniques [6, 43, 92, 88] into cross-modal and cross-frame fusion to improve the model's contextual representation. Moreover, we could introduce extra edge signals [65, 28] to aid the model in capturing object boundary details. In addition, it could be beneficial to explore more sophisticated fusion techniques [16, 95, 25] to promote effective interaction between multimodal and temporal information.
(2) Efficiency: Although the engagement of RGB-D videos brings significant improvement, it introduces additional model parameters. More lightweight strategies can be explored to improve efficiency. For instance, we can develop lightweight operations such as depthwise separable convolution [10] or neural architecture search techniques [84], to advance feature extractors. On the other hand, we could explore knowledge distillation schemes [64, 54] to transfer depth structure knowledge to the RGB stream, thereby reducing the heavy overhead of the depth encoder. To evaluate the impact of lightweight strategy, we conduct an exploratory experiment by replacing the conventional convolution in our network encoder with the more efficient depth-wise separable convolution. Our experiments show a reduction in network parameters from 97.3M to 58.1M and memory usage from 17.15G to 16.21G, along with an increase in inference speed from 16.7 FPS to 21.5 FPS. However, this change leads to a drop in performance, with the F-measure falling from 0.610 to 0.575. These findings underscore the potential of efficiency-enhancing strategies, and also highlight the need to balance accuracy with efficiency in saliency model design.
(3) Temporal Modeling: In this work, we conduct a preliminary investigation on the benefits of RGB-D videos for saliency detection by incorporating an well-designed temporal aggregation scheme [57] that focuses on modeling long-term dependencies. As a potential extension, we could consider introducing optical flow to capture short-term temporal cues. However, it's important to note that directly integrating optical flow could increase training complexity and potentially introduce noise, especially in complex scenes. Meanwhile, it is wroth exploring the saliency change problem (e.g., the salient object changing from a person to a table) when referring to temporal information of previous frames. In the DVSOD baseline, we attempt to address it through engaging the attention mechanism within [57], which selectively discounts less relevant information and fetches the most relevant features. We encourage further investigation on this matter, and believe there is a significant opportunity for the development of more effective temporal modeling schemes that can address these challenges.
(4) Dataset: As the first dataset proposed for DVSOD, there is still considerable room for improving the various aspects of the dataset such as quality, quantity, and diversity by involving a broader range of scenarios of extreme lighting conditions, as well as various types of occlusions. Furthermore, annotating such large-scale RGB-D video dataset at pixel-level is very costly in terms of both time and effort. We may explore the recent advances of photorealistic rendering techniques [11, 20] to simulate data or supplement annotations to help address challenges within the dataset.
(5) Evaluation Metrics: Evaluation metrics are crucial for model training, testing, and benchmarking. However, the saliency detection community largely relies on classic metrics such as MAE and F-measure, which are not designed to assess sequential DVSOD tasks specifically. Meanwhile, the Temporal Coherence (TC) metric, frequently used in evaluating RGB video, may not accurately