Promoting Saliency From Depth: Deep Unsupervised RGB-D Saliency Detection

1 INTRODUCTION . The recent development of RGB-D salient object detection , i.e. , RGB-D SOD , is especially fueled by the increasingly accessible 3D imaging sensors ( Giancola et al. , 2018 ) and their diverse applications , including image caption generation ( Xu et al. , 2015 ) , image retrieval ( Ko et al. , 2004 ; Shao & Brady , 2006 ) and video analysis ( Liu et al. , 2008 ; Wang et al. , 2018 ) , to name a few . Provided with multimodality input of an RGB image and its depth map , the task of RGB-D SOD is to effectively identify and segment the most distinctive objects in a scene . The state-of-the-art RGB-D SOD approaches ( Li et al. , 2021a ; Ji et al. , 2020b ; Chen et al. , 2020b ; Li et al. , 2020b ; 2021b ) typically entail an image-to-mask mapping pipeline that is based on the powerful deep learning paradigms of e.g. , VGG16 ( Simonyan & Zisserman , 2015 ) or ResNet50 ( He et al. , 2016 ) . This strategy has led to excellent performance . On the other hand , these RGB-D SOD methods are fully supervised , thus demand a significant amount of pixel-level training annotations . This however becomes much less appealing in practical scenarios , owing to the laborious and time-consuming process in obtaining manual annotations . It is therefore natural and desirable to contemplating unsupervised alternatives . Unfortunately , existing unsupervised RGB-D SOD methods , such as global priors ( Ren et al. , 2015 ) , center prior ( Zhu et al. , 2017b ) , and depth contrast prior ( Ju et al. , 2014 ) , rely primarily on handcrafted feature representations . This is in stark contrast to the deep representations learned by their supervised SOD counterparts , which in effect imposes severe limitations on the feature representation power that may otherwise benefit greatly from the potentially abundant unlabeled RGB-D images . These observations motivate us to explore a new problem of deep unsupervised RGB-D saliency detection : given an unlabeled set of RGB-D images , deep neural network is trained to predict saliency without any laborious human annotations in the training stage . A relatively straightforward idea is to exploit the outputs from traditional RGB-D method as pseudo-labels , which are internally employed ∗Corresponding author Published as a conference paper at ICLR 2022 SE DSR CDCP GP DES DCMC MST ACSD LHM BSCA RBD CDB Baseline 121086420 30 40 50 60 Inference time ( s ) W ei gh te d F -m ea su re A cc ur ac y ( % ) RGB-based method RGB & D-based method Ours +37 % Unlabeled/Labelded Training Data： RGB Depth GT Tradition，Baseline，Ours Initial Label Baseline Ours RGB Image Ground Truth ( a ) Deep unsupervised RGB-D saliency detection ( b ) Accuracy and inference speed comparison 5_06-39-00.png，NLPR Depth Map Figure 1 : ( a ) An illustration of deep unsupervised RGB-D saliency detection . ‘ Initial label ’ is generated by a traditional method . ‘ Baseline ’ shows the saliency map generated by saliency network trained with initial pseudo-labels . ‘ Ours ’ shows our final results . ( b ) Efficiency and effectiveness comparison over a wide range of unsupervised SOD methods on the NLPR benchmark . Promoting Saliency From Depth : our approach achieves a large-margin improvement over the baseline , by engaging depth information to improve pseudo-labels in the training process , without introducing additional computational cost during inference , shown in red arrow . to train the saliency prediction network ( ‘ baseline ’ ) . Moreover , the input depth map may serve as a complementary source of information in refining the pseudo-labels , as it contains cues of spatial scene layout that may help in exposing the salient objects . Nevertheless , practical examination reveals two main issues : ( 1 ) Inconsistency and large variations in raw depth maps : as illustrated in Fig . 1 ( a ) , similar depth values are often shared by a salient object and its surrounding , making it very difficult in extracting the salient regions from depth without explicit pixel-level supervision ; ( 2 ) Noises from unreliable pseudo-labels : unreliable pseudo-labels may inevitably bring false positive into training , resulting in severe damage in its prediction performance . To address the above challenges , the following two key components are considered in our approach . First , a depth-disentangled saliency update ( DSU ) framework is proposed to iteratively refine & update the pseudo-labels by engaging the depth knowledge . Here a depth-disentangled network is devised to explicitly learn the discriminative saliency cues and non-salient background from raw depth map , denoted as saliency-guided depth DSal and non-saliency-guided depth DNonSal , respectively . This is followed by a depth-disentangled label update ( DLU ) module that takes advantage of DSal to emphasize saliency response from pseudo-label ; it also utilizes DNonSal to eliminate the background influence , thus facilitating more trustworthy supervision signals in training the saliency network . Note the DSU module is not engaged at test time . Therefore , at test time , our trained model takes as input only an RGB image , instead of involving both RGB and depth as input and in the follow-up computation . Second , an attentive training strategy is introduced to alleviate the issue of noisy pseudo-labels ; it is achieved by re-weighting the training samples in each training batch to focus on those more reliable pseudo-labels . As demonstrated in Fig . 1 ( b ) , our approach works effectively and efficiently in practice . It significantly outperforms existing unsupervised SOD methods on the widely-used NLPR benchmark . Specifically , it improves over the baseline by 37 % , a significant amount without incurring extra computation cost . Besides , the test time execution of our approach is at 35 frame-per-second ( FPS ) , the fastest among all RGB-D unsupervised methods , and on par with the most efficient RGB-based methods . In summary , our main contributions are as follows : • To our knowledge , our work is the first in exploring deep representation to tackle the problem of unsupervised RGB-D saliency detection . This is enabled by two key components in the training process , namely the DSU strategy to produce & refine pseudo-labels , and the attentive training strategy to alleviate the influence of noisy pseudo-labels . It results in a light-weight architecture that engages only RGB data at test time ( i.e. , w/o depth map ) , achieving a significant improvement without extra computation cost . • Empirically , our approach outperforms state-of-the-art unsupervised methods on four public benchmarks . Moreover , it runs in real time at 35 FPS , much faster than existing unsupervised RGB-D SOD methods , and at least on par with the fastest RGB counterparts . • Our approach could be adapted to work with fully-supervised scenario . As demonstrated in Sec . 4.4 , augmented with our proposed DSU module , the empirical results of existing RGB-D SOD models have been notably improved . Published as a conference paper at ICLR 2022 2 RELATED WORK . Remarkable progresses have been made recently in salient object detection ( Ma et al. , 2021 ; Xu et al. , 2021 ; Pang et al. , 2020b ; Zhao et al. , 2020b ; Tsai et al. , 2018 ; Liu et al. , 2018 ) , where the performance tends to deteriorate in the presence of cluttered or low-contrast backgrounds . Moreover , promising results are shown by a variety of recent efforts ( Chen et al. , 2021a ; Luo et al. , 2020 ; Chen et al. , 2020a ; Zhao et al. , 2022 ; Desingh et al. , 2013 ; Li et al. , 2020a ; Liao et al. , 2020 ; Shigematsu et al. , 2017 ) that integrate effective depth cues to tackle these issues . Those methods , however , typically demand extensive annotations , which are labor-intensive and time-consuming . This naturally leads to the consideration of unsupervised SODs that do not rely on such annotations . Prior to the deep learning era ( Wang et al. , 2021 ; Zhao et al. , 2020a ) , traditional RGB-D saliency methods are mainly based on the manually-crafted RGB features and depth cues to infer a saliency map . Due to their lack of reliance on manual human annotation , these traditional methods can be regarded as early manifestations of unsupervised SOD . Ju et al . ( Ju et al. , 2014 ) consider the incorporation of a prior induced from anisotropic center-surround operator , and an additional depth prior . Ren et al . ( Ren et al. , 2015 ) also introduce two priors : normalized depth prior and the globalcontext surface orientation prior . In contrary to direct utilization of the depth contrast priors , local background enclosure prior is explicitly developed by ( Feng et al. , 2016 ) . Interested readers may refer to ( Fan et al. , 2020a ; Peng et al. , 2014 ; Zhou et al. , 2021 ) for more in-depth surveys and analyses . However , by relying on manually-crafted priors , these methods tend to have inferior performance . Meanwhile , considerable performance gain has been achieved by recent efforts in RGB-based unsupervised SOD ( Zhang et al. , 2017 ; 2018 ; Nguyen et al. , 2019 ; Hsu et al. , 2018a ) , which instead construct automated feature representations using deep learning . A typical strategy is to leverage the noisy output produced by traditional methods as pseudo-label ( i.e. , supervisory signal ) for training saliency prediction net . The pioneering work of Zhang et al . ( Zhang et al. , 2017 ) fuses the outputs of multiple unsupervised saliency models as guiding signals in CNN training . In ( Zhang et al. , 2018 ) , competitive performance is achieved by fitting a noise modeling module to the noise distribution of pseudo-label . Instead of directly using pseudo-labels from handcrafted methods , Nguyen et al . ( Nguyen et al. , 2019 ) further refine pseudo-labels via a self-supervision iterative process . Besides , deep unsupervised learning has been considered by ( Hsu et al. , 2018b ; Tsai et al. , 2019 ) for the co-saliency task , where superior performance has been obtained by engaging a fusion-learning scheme ( Hsu et al. , 2018b ) , or utilizing dedicated loss terms ( Tsai et al. , 2019 ) . These methods demonstrate that the incorporation of powerful deep neural network brings better feature representation than those unsupervised SOD counterparts based on handcrafted features . In this work , a principled research investigation on deep unsupervised RGB-D saliency detection . is presented for the first time . Different from existing deep RGB-based unsupervised SOD ( Zhang et al. , 2017 ; 2018 ) that use pseudo-labels from the handcrafted methods directly , our work aims to refine and update pseudo-labels to remove the undesired noise . And compared to the refinement method ( Nguyen et al. , 2019 ) using self-supervision technique , our approach instead adopts disentangled depth information to promote pseudo-labels . 3 METHODOLOGY . 3.1 OVERVIEW . Fig . 2 presents an overview of our Depth-disentangled Saliency Update ( DSU ) framework . Overall our DSU strives to significantly improve the quality of pseudo-labels , that leads to more trustworthy supervision signals for training the saliency network . It consists of three key components . First is a saliency network responsible for saliency prediction , whose initial supervision signal is provided by traditional handcrafted method without using human annotations . Second , a depth network and a depth-disentangled network are designed to decompose depth cues into saliency-guided depth DSal and non-saliency-guided depth DNonSal , to explicitly depict saliency cues in the spatial layout . Third , a depth-disentangled label update ( DLU ) module is devised to refine and update the pseudolabels , by engaging the learned DSal and DNonSal . The updated pseudo-labels could in turn provide more trustworthy supervisions for the saliency network . Moreover , an attentive training strategy ( ATS ) is incorporated when training the saliency network , tailored for noisy unsupervised learning by mitigating the ambiguities caused by noisy pseudo-labels . We note that the DSU and ATS are not performed at test time , so does not affect the inference speed . In other words , the inference stage In this paper , unsupervised learning refers to learning without using human annotation ( Zhang et al. , 2017 ) . Published as a conference paper at ICLR 2022 2021.06.15 f r overall NJUD : 001081 involves only the black dashed portion of the proposed network architecture in Fig . 2 , i.e. , only RGB images are used for predicting saliency , which enables very efficient detection .

Summary: The paper presents an unsupervised method for RGBD SOD. It internally generates pseudo labels for training by extracting and refining/updating handcrafted features under the framework of ‘depth-disentangling’. Since the updating module can be removed during the inference, it achieves high efficiency.  The method also demonstrates SOTA performance on several benchmarks.