Promoting Saliency From Depth: Deep Unsupervised RGB-D Saliency Detection

1 INTRODUCTION . The recent development of RGB-D salient object detection , i.e. , RGB-D SOD , is especially fueled by the increasingly accessible 3D imaging sensors ( Giancola et al. , 2018 ) and their diverse applications , including image caption generation ( Xu et al. , 2015 ) , image retrieval ( Ko et al. , 2004 ; Shao & Brady , 2006 ) and video analysis ( Liu et al. , 2008 ; Wang et al. , 2018 ) , to name a few . Provided with multimodality input of an RGB image and its depth map , the task of RGB-D SOD is to effectively identify and segment the most distinctive objects in a scene . The state-of-the-art RGB-D SOD approaches ( Li et al. , 2021a ; Ji et al. , 2020b ; Chen et al. , 2020b ; Li et al. , 2020b ; 2021b ) typically entail an image-to-mask mapping pipeline that is based on the powerful deep learning paradigms of e.g. , VGG16 ( Simonyan & Zisserman , 2015 ) or ResNet50 ( He et al. , 2016 ) . This strategy has led to excellent performance . On the other hand , these RGB-D SOD methods are fully supervised , thus demand a significant amount of pixel-level training annotations . This however becomes much less appealing in practical scenarios , owing to the laborious and time-consuming process in obtaining manual annotations . It is therefore natural and desirable to contemplating unsupervised alternatives . Unfortunately , existing unsupervised RGB-D SOD methods , such as global priors ( Ren et al. , 2015 ) , center prior ( Zhu et al. , 2017b ) , and depth contrast prior ( Ju et al. , 2014 ) , rely primarily on handcrafted feature representations . This is in stark contrast to the deep representations learned by their supervised SOD counterparts , which in effect imposes severe limitations on the feature representation power that may otherwise benefit greatly from the potentially abundant unlabeled RGB-D images . These observations motivate us to explore a new problem of deep unsupervised RGB-D saliency detection : given an unlabeled set of RGB-D images , deep neural network is trained to predict saliency without any laborious human annotations in the training stage . A relatively straightforward idea is to exploit the outputs from traditional RGB-D method as pseudo-labels , which are internally employed ∗Corresponding author Published as a conference paper at ICLR 2022 SE DSR CDCP GP DES DCMC MST ACSD LHM BSCA RBD CDB Baseline 121086420 30 40 50 60 Inference time ( s ) W ei gh te d F -m ea su re A cc ur ac y ( % ) RGB-based method RGB & D-based method Ours +37 % Unlabeled/Labelded Training Data： RGB Depth GT Tradition，Baseline，Ours Initial Label Baseline Ours RGB Image Ground Truth ( a ) Deep unsupervised RGB-D saliency detection ( b ) Accuracy and inference speed comparison 5_06-39-00.png，NLPR Depth Map Figure 1 : ( a ) An illustration of deep unsupervised RGB-D saliency detection . ‘ Initial label ’ is generated by a traditional method . ‘ Baseline ’ shows the saliency map generated by saliency network trained with initial pseudo-labels . ‘ Ours ’ shows our final results . ( b ) Efficiency and effectiveness comparison over a wide range of unsupervised SOD methods on the NLPR benchmark . Promoting Saliency From Depth : our approach achieves a large-margin improvement over the baseline , by engaging depth information to improve pseudo-labels in the training process , without introducing additional computational cost during inference , shown in red arrow . to train the saliency prediction network ( ‘ baseline ’ ) . Moreover , the input depth map may serve as a complementary source of information in refining the pseudo-labels , as it contains cues of spatial scene layout that may help in exposing the salient objects . Nevertheless , practical examination reveals two main issues : ( 1 ) Inconsistency and large variations in raw depth maps : as illustrated in Fig . 1 ( a ) , similar depth values are often shared by a salient object and its surrounding , making it very difficult in extracting the salient regions from depth without explicit pixel-level supervision ; ( 2 ) Noises from unreliable pseudo-labels : unreliable pseudo-labels may inevitably bring false positive into training , resulting in severe damage in its prediction performance . To address the above challenges , the following two key components are considered in our approach . First , a depth-disentangled saliency update ( DSU ) framework is proposed to iteratively refine & update the pseudo-labels by engaging the depth knowledge . Here a depth-disentangled network is devised to explicitly learn the discriminative saliency cues and non-salient background from raw depth map , denoted as saliency-guided depth DSal and non-saliency-guided depth DNonSal , respectively . This is followed by a depth-disentangled label update ( DLU ) module that takes advantage of DSal to emphasize saliency response from pseudo-label ; it also utilizes DNonSal to eliminate the background influence , thus facilitating more trustworthy supervision signals in training the saliency network . Note the DSU module is not engaged at test time . Therefore , at test time , our trained model takes as input only an RGB image , instead of involving both RGB and depth as input and in the follow-up computation . Second , an attentive training strategy is introduced to alleviate the issue of noisy pseudo-labels ; it is achieved by re-weighting the training samples in each training batch to focus on those more reliable pseudo-labels . As demonstrated in Fig . 1 ( b ) , our approach works effectively and efficiently in practice . It significantly outperforms existing unsupervised SOD methods on the widely-used NLPR benchmark . Specifically , it improves over the baseline by 37 % , a significant amount without incurring extra computation cost . Besides , the test time execution of our approach is at 35 frame-per-second ( FPS ) , the fastest among all RGB-D unsupervised methods , and on par with the most efficient RGB-based methods . In summary , our main contributions are as follows : • To our knowledge , our work is the first in exploring deep representation to tackle the problem of unsupervised RGB-D saliency detection . This is enabled by two key components in the training process , namely the DSU strategy to produce & refine pseudo-labels , and the attentive training strategy to alleviate the influence of noisy pseudo-labels . It results in a light-weight architecture that engages only RGB data at test time ( i.e. , w/o depth map ) , achieving a significant improvement without extra computation cost . • Empirically , our approach outperforms state-of-the-art unsupervised methods on four public benchmarks . Moreover , it runs in real time at 35 FPS , much faster than existing unsupervised RGB-D SOD methods , and at least on par with the fastest RGB counterparts . • Our approach could be adapted to work with fully-supervised scenario . As demonstrated in Sec . 4.4 , augmented with our proposed DSU module , the empirical results of existing RGB-D SOD models have been notably improved . Published as a conference paper at ICLR 2022 2 RELATED WORK . Remarkable progresses have been made recently in salient object detection ( Ma et al. , 2021 ; Xu et al. , 2021 ; Pang et al. , 2020b ; Zhao et al. , 2020b ; Tsai et al. , 2018 ; Liu et al. , 2018 ) , where the performance tends to deteriorate in the presence of cluttered or low-contrast backgrounds . Moreover , promising results are shown by a variety of recent efforts ( Chen et al. , 2021a ; Luo et al. , 2020 ; Chen et al. , 2020a ; Zhao et al. , 2022 ; Desingh et al. , 2013 ; Li et al. , 2020a ; Liao et al. , 2020 ; Shigematsu et al. , 2017 ) that integrate effective depth cues to tackle these issues . Those methods , however , typically demand extensive annotations , which are labor-intensive and time-consuming . This naturally leads to the consideration of unsupervised SODs that do not rely on such annotations . Prior to the deep learning era ( Wang et al. , 2021 ; Zhao et al. , 2020a ) , traditional RGB-D saliency methods are mainly based on the manually-crafted RGB features and depth cues to infer a saliency map . Due to their lack of reliance on manual human annotation , these traditional methods can be regarded as early manifestations of unsupervised SOD . Ju et al . ( Ju et al. , 2014 ) consider the incorporation of a prior induced from anisotropic center-surround operator , and an additional depth prior . Ren et al . ( Ren et al. , 2015 ) also introduce two priors : normalized depth prior and the globalcontext surface orientation prior . In contrary to direct utilization of the depth contrast priors , local background enclosure prior is explicitly developed by ( Feng et al. , 2016 ) . Interested readers may refer to ( Fan et al. , 2020a ; Peng et al. , 2014 ; Zhou et al. , 2021 ) for more in-depth surveys and analyses . However , by relying on manually-crafted priors , these methods tend to have inferior performance . Meanwhile , considerable performance gain has been achieved by recent efforts in RGB-based unsupervised SOD ( Zhang et al. , 2017 ; 2018 ; Nguyen et al. , 2019 ; Hsu et al. , 2018a ) , which instead construct automated feature representations using deep learning . A typical strategy is to leverage the noisy output produced by traditional methods as pseudo-label ( i.e. , supervisory signal ) for training saliency prediction net . The pioneering work of Zhang et al . ( Zhang et al. , 2017 ) fuses the outputs of multiple unsupervised saliency models as guiding signals in CNN training . In ( Zhang et al. , 2018 ) , competitive performance is achieved by fitting a noise modeling module to the noise distribution of pseudo-label . Instead of directly using pseudo-labels from handcrafted methods , Nguyen et al . ( Nguyen et al. , 2019 ) further refine pseudo-labels via a self-supervision iterative process . Besides , deep unsupervised learning has been considered by ( Hsu et al. , 2018b ; Tsai et al. , 2019 ) for the co-saliency task , where superior performance has been obtained by engaging a fusion-learning scheme ( Hsu et al. , 2018b ) , or utilizing dedicated loss terms ( Tsai et al. , 2019 ) . These methods demonstrate that the incorporation of powerful deep neural network brings better feature representation than those unsupervised SOD counterparts based on handcrafted features . In this work , a principled research investigation on deep unsupervised RGB-D saliency detection . is presented for the first time . Different from existing deep RGB-based unsupervised SOD ( Zhang et al. , 2017 ; 2018 ) that use pseudo-labels from the handcrafted methods directly , our work aims to refine and update pseudo-labels to remove the undesired noise . And compared to the refinement method ( Nguyen et al. , 2019 ) using self-supervision technique , our approach instead adopts disentangled depth information to promote pseudo-labels . 3 METHODOLOGY . 3.1 OVERVIEW . Fig . 2 presents an overview of our Depth-disentangled Saliency Update ( DSU ) framework . Overall our DSU strives to significantly improve the quality of pseudo-labels , that leads to more trustworthy supervision signals for training the saliency network . It consists of three key components . First is a saliency network responsible for saliency prediction , whose initial supervision signal is provided by traditional handcrafted method without using human annotations . Second , a depth network and a depth-disentangled network are designed to decompose depth cues into saliency-guided depth DSal and non-saliency-guided depth DNonSal , to explicitly depict saliency cues in the spatial layout . Third , a depth-disentangled label update ( DLU ) module is devised to refine and update the pseudolabels , by engaging the learned DSal and DNonSal . The updated pseudo-labels could in turn provide more trustworthy supervisions for the saliency network . Moreover , an attentive training strategy ( ATS ) is incorporated when training the saliency network , tailored for noisy unsupervised learning by mitigating the ambiguities caused by noisy pseudo-labels . We note that the DSU and ATS are not performed at test time , so does not affect the inference speed . In other words , the inference stage In this paper , unsupervised learning refers to learning without using human annotation ( Zhang et al. , 2017 ) . Published as a conference paper at ICLR 2022 2021.06.15 f r overall NJUD : 001081 involves only the black dashed portion of the proposed network architecture in Fig . 2 , i.e. , only RGB images are used for predicting saliency , which enables very efficient detection .

Summary: This paper presents an unsupervised RGB saliency detection model to learn from the pseudo-labels given by the traditional handcrafted approach. To enhance the quality of the pseudo labels, the author proposes to disentangle the depth data to promote the saliency signal and compress undesired noise. Last but not least, an attentive training strategy is integrated to teach the network from iteratively updating supervisory labels. Experimental results demonstrate the effectiveness of the proposed depth-disentangled saliency update framework and can help improve the performance of on-the-shelf approaches.