Joint Semantic Mining for Weakly Supervised RGB-D Salient Object Detection

1 Introduction . As a fundamental computer vision task , salient object detection ( SOD ) aims at locating and segmenting visually distinctive objects in a scene . It plays an important role in a variety of downstream applications including image retrieval [ 31 , 65 ] , medical analysis [ 48 , 28 , 25 ] , multimodal fusion [ 79 , 80 ] and video analysis [ 88 , 81 , 90 ] . Recent progress in supervised RGB-D SOD [ 5 , 52 , 10 , 26 , 89 , 33 ] has demonstrated significant benefits of engaging depth information in saliency detection from complex scenes . The success of these fully-supervised methods , however , relies heavily on the large-scale , precise , pixel-level annotations , which are often laborious and time-consuming to acquire . On the other hand , an image usually comes with additional information in its meta-data such as tags and captions from users to describe the scene context and content , which may serve as cheap weak-supervision signals . These weak supervision signals are nonetheless noisy and have mixed qualities . Similar weak-supervision signals have been explored in RGB-image based SOD [ 63 , 70 , 38 ] , where the noisy nature of these side information is unfortunately overlooked . To further complicate the matter , the lack of explicit pixel-level supervision brings new challenge to the RGB-D SOD task : the depth values from raw depth maps are often noisy and sometimes inconsistent . For example , in Fig . 1 , * means equal contributions . Wei Ji ( wji3 @ ualberta.ca ) is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) 000220_left similar depth values are shared by the cat and the underneath couch , making it difficult to discern the salient object from backgrounds . Without the explicit pixel-level supervision , existing cross-modal fusion strategies adopted by fully-supervised RGB-D methods [ 36 , 55 , 37 , 27 ] would simply fail . These observations motivate us to consider the new problem of weakly-supervised RGB-D salient object detection , which takes as input the RGB and depth images , as well as weak supervision signals such as image-level tags and captions , as illustrated in Fig . 1 . By removing the demand for laborious per-pixel annotations , it also brings new challenges : 1 ) how to address the noisy nature of the weak supervision signals ; 2 ) how to tackle the depth noise and inconsistency to facilitate proper separation of foreground and background regions . This leads us to propose the use of pseudo-labels with iterative refinements in training : the pseudolabel provides internal pixel-level supervision signals , which is progressively updated by reconciling the multimodal input signals and the current information flow of the neural net , based on the previous pseudo-label . As illustrated in Fig . 2 , this is realized by an interaction between two core modules , namely spatial semantic modeling ( SSM ) and joint semantic mining ( JSM ) : SSM is designed to capture the saliency-specific depth semantics , to eliminate the background noises in the coarse saliency prediction , and to generate a depth-refined pseudo-label . This simple yet effective module is very generic , which could be easily plugged-in different setups , including unsupervised & fullysupervised scenarios ; meanwhile , the JSM module is proposed to leverage depth semantics and weak supervision signals for attaining more reliable pseudo-labels . Specifically , a partial textual input , i.e. , image-level tag and caption with its salient word being masked , is fed into a dedicated textual semantic modeling or TSM to estimate the confidence scores of competing pseudo-labels , and to fill-in-the-blank . Intuitively , a semantically consistent pseudo-label should provide better context cues to reconstruct the salient word ; while a closer guess of the masked word would indicate a better pseudo-label . The alternation between SSM and JSM modules is thus expected to give rise to more trustworthy pseudo-labels . At test time , it is then sufficient to take an input RGB image and activate a light-weight network to deliver its final prediction . That is , test time input involves only an RGB image , without the need of any depth map or image-level tags and captions . This drastically simplifies the input requirement and reduces the computation burden at deployment stage . Moreover , given the lack of training dataset for the weakly-supervised setting , we adapt existing RGB-D training dataset to annotate additional image-level tags and captions , which is referred to as the CapS dataset . The main contributions of this paper are as follows . ( 1 ) A new problem of weakly-supervised RGB-D salient object detection is considered . In this regard , a CapS dataset is curated by augmenting the existing RGB-D SOD training dataset with image-level tagging and captioning annotations . ( 2 ) The key ingredient of our approach involves the production of pseudo-labels with iterative refinements , realized by iterative updates between two internal modules , SSM and JSM . Empirical experiments demonstrate the effectiveness of our approach in weakly-supervised setting . Moreover , ( 3 ) after proper adaptation of our approach in unsupervised and fully-supervised scenarios , superior performance is also observed when comparing to the respective state-of-the-art methods . ( 4 ) Our test time inference amounts to executing a light-weight saliency network : as illustrated in dotted line at Fig . 2 , only an RGB image is used as its input , thus allows efficient and effective inference . 2 Related Work . RGB-D salient object detection [ 87 , 93 , 94 ] has been an active field of research in the past few years , where the incorporation of depth cues has been demonstrated [ 13 , 34 , 82 , 75 , 15 , 35 , 60 , 43 , 85 ] to improve performance especially in complex scenes . Existing RGB-D methods aim to design effective feature fusion strategies for learning representative cross-modal features . Typically , Chen et al . [ 8 ] employ two-stream CNNs-based models and perform fusion by adding or concatenating paired features at shallow or deep layers . Fu et al . [ 19 ] utilize a Siamese network to jointly learn RGB and depth inputs for mining useful complementary features . To promote multi-modal interactions , Li et al . [ 37 ] design a cross-modal weighting strategy to encourage comprehensive interactions between RGB and depth information . However , those methods unfortunately rely on costly pixellevel annotations , which are tedious and time-consuming to acquire . This motivates us to consider a weakly-supervised approach . In what follows , our focus will be mainly toward related weaklysupervised methods developed for saliency detection from RGB images , where the differences of our approach from existing methods would be clarified . Instead of using costly pixel-level annotations , some recent efforts instead explore cheap alternatives such as image-level tags ( categories ) [ 38 , 63 , 71 , 46 , 2 ] , image captions [ 70 ] , scribble labels [ 69 , 76 ] , and noisy pseudo-labels from handcrafted methods [ 45 , 72 , 73 , 77 , 68 , 47 , 67 , 49 , 66 ] . The pioneering work [ 63 ] leverage image-level tags or categories that could be augmented onto existing large-scale dataset at low-cost . The same scenario is also considered by Li et al . [ 38 ] , where a composite pipeline combining graphical model with CNNs is designed . The trained network however tends to highlight the most discriminative region instead of the intended salient object out of the scene due to the sparse image-level supervisions . Image captions are examined by Zeng et al . [ 70 ] as supervision input ; in their work image classification network and caption generation network are jointly trained to obtain pseudo-labels , which achieves descent performance . Scribble is another type of weak supervision signal , where a tiny fraction of image pixels are labeled by users as being foreground or background . Due to the annotation sparsity , object structure and details can not be easily inferred . Zhang et al . [ 76 ] introduce a gated structure-aware loss as well as an auxiliary edge detection network to uncover the complete object . Meanwhile , Yu et al . [ 69 ] explore self-consistency among multi-scale outputs and design a local coherence loss to propagate the labels to unlabeled regions based on image features , thus enabling the detection of objects with smooth textures . However , existing weakly supervised saliency methods are solely based on RGB image . Unlike the prevalence of fully-supervised RGB-D SOD , it has never been considered the incorporation of depth cues in existing literature . This leads us to address this problem in the presence of image tags and captions as weak supervision signals . Different from existing methods [ 38 , 24 , 63 , 70 ] that train image classification networks or caption generation networks to delineate potential salient regions , masked salient word in captions are to be reconstructed in our work by leveraging visual saliency features . This is then used to estimate the confidence scores of pseudo-labels . 3 Methodology . 3.1 The Overall Architecture . An overview of our approach is illustrated in Fig . 2 . It consists of a saliency network responsible for saliency prediction , a spatial ( depth ) semantic modeling ( SSM ) to generate depth-refined pseudo-label , a textual ( caption ) semantic modeling ( TSM ) to estimate the confidences of different pseudo-labels , and a joint semantic mining ( JSM ) strategy to refine & update pseudo-label . Overall our pipeline aims to gradually improve the quality of noisy pseudo-labels by jointly mining the useful spatial semantics from the depth map and textual semantics from the tags and captions , to produce more trustworthy supervision signals , which in turns results in better training of the saliency network . Specifically , the popular encoder-decoder architecture [ 64 ] in SOD is adopted in both saliency network and depth network . Initial supervision signal for the saliency network is provided by traditional handcrafted methods . The predicted saliency map , together with the raw depth map , are processed to generate the saliency-guided spatial supervision signal for the depth network . Then the predicted saliency-oriented depth semantics is utilized to eliminate background noises ( non-salient regions ) in coarse prediction , and to generate a depth-refined pseudo-label . This is followed by our JSM strategy , which takes in the image-level tags and captions through TSM to estimate the confidence scores of pseudo-labels ; updated pseudo-label is then formed based on the confidence-weighted depth-refined pseudo-label and current pseudo-label , which provides more trustworthy supervision signal for the saliency network . Note our test time inference involves only the black dashed portion in Fig . 2 , which takes as input only the RGB image , thus enables efficient saliency prediction . 3.2 Saliency-oriented Spatial ( Depth ) Semantic Modeling . Initial pseudo-labels are generated by traditional saliency models , which often contain excessive noise . As illustrated in Fig . 3 , our spatial semantic modeling ( SSM ) is to produce a more reliable depth-refined pseudo-label , achieved by explicitly capturing saliency-specific depth semantics from the depth map to eliminate possible background noise in the coarse saliency prediction . Concretely , during training , we first generate a saliency-guided depth mask Dmask by multiplying the rough saliency prediction Spred with the raw depth map Dmap in a spatial attention manner . Here , a Gaussian smooth operation is applied to smooth the predicted saliency area , to effectively perceive and capture more saliency areas from depth . The procedure is formulated as : Dmask = Ωmax ( Fgauss ( Spred , k ) , Spred ) ⊗Dmap , ( 1 ) where Fgauss ( · , k ) indicates a convolution operation with Gaussian kernel k and zero bias ; Ωmax ( · , · ) is a maximum function to preserve the higher values between the smoothed and the original maps . ⊗ is element-wise multiplication . In this paper , the size and standard deviation of kernel k are learnable through the model training procedure and are initialized with values 32 and 4 , respectively . After obtaining Dmask , a depth network is trained to learn the saliency-specific depth semantics DS , using the mean square error ( MSE ) loss function . The internal inspection evidences in Fig . 4 suggest that DS ( depth semantics , 6th column ) is able to capture discriminative saliency cues from the raw depth map under the supervision of Dmask ( 5th column ) . In addition , the learned DS can be further processed to generate the depth-refined pseudo-labels . This procedure is only employed when performing pseudo-label update . Specifically , we feed DS into an Background Noise Suppression block to help eliminate the background noises ( non-salient 1_02-02-40.jpg ; 000489_left.png ; 000423_left1.png regions ) in the coarse prediction Spred . In this block , a reverse operation is first employed on DS to highlight background regions by 1−DS . This is followed by the pixel-wise subtraction to suppress the non-salient negative responses in Spred . Finally , a ReLU function and a normalization procedure are adopted to suppress negative numbers and normalize the result to the range of [ 0 , 1 ] . This procedure of obtaining the depth-refined pseudo-label SD could be expressed by Si , jD = Si , jd −min ( Sd ) max ( Sd ) −min ( Sd ) + ε , Sd = ReLU ( Spred−λd ( 1−DS ) ) , i ∈ [ 1 , H ] , j ∈ [ 1 , W ] . ( 2 ) Here λd ∈ [ 0 , 1 ] is a constant to control the degree of the subtracted background noises and avoid negatively suppressing salient regions that have relatively low saliency scores in SD . Throughout experiments , λd is empirically set to 0.5 , and ε to 1e-5 . H and W are the height and width of the input image , respectively . The pseudo-label refinement dynamics are visually inspected in Fig . 4 , while Fig . 8 presents the corresponding quantitative error analysis over iterations . Empirical evidence suggests as the training proceeds , quality of the pseudo-label is significantly improved . Moreover , at this stage , we can directly utilize the depth-refined label SD to update pseudo-labels ( as in Fig . 6 ( b ) ) . This can be referred as unsupervised RGB-D SOD since no weak labels are used .

Summary: This article addresses salient object detection using depth maps and image captions as supervised signals. The proposed network includes a SSM for spatial semantic modelling and a TSM for cross-modal semantic mining. Additionally, a CapS dataset is built to extend existing datasets with extra weak annotations.