Joint Semantic Mining for Weakly Supervised RGB-D Salient Object Detection
Authors: Jingjing Li
Wei Ji
Qi Bi
Cheng Yan
Miao Zhang
Yongri Piao
Huchuan Lu
Li Cheng
Venue: NeurIPS 2021 Poster (2021)
Keywords: Salient Object Detection
Saliency Detection
Weakly Supervised RGB-D Salient Object Detection

Abstract: Training saliency detection models with weak supervisions, e.g., image-level tags or captions, is appealing as it removes the costly demand of per-pixel annotations. Despite the rapid progress of RGB-D saliency detection in fully-supervised setting, it however remains an unexplored territory when only weak supervision signals are available. This paper is set to tackle the problem of weakly-supervised RGB-D salient object detection. The key insight in this effort is the idea of maintaining per-pixel pseudo-labels with iterative refinements by reconciling the multimodal input signals in our joint semantic mining (JSM). Considering the large variations in the raw depth map and the lack of explicit pixel-level supervisions, we propose spatial semantic modeling (SSM) to capture saliency-specific depth cues from the raw depth and produce depth-refined pseudo-labels. Moreover, tags and captions are incorporated via a fill-in-the-blank training in our textual semantic modeling (TSM) to estimate the confidences of competing pseudo-labels. At test time, our model involves only a light-weight sub-network of the training pipeline, i.e., it requires only an RGB image as input, thus allowing efficient inference. Extensive evaluations demonstrate the effectiveness of our approach under the weakly-supervised setting. Importantly, our method could also be adapted to work in both fully-supervised and unsupervised paradigms. In each of these scenarios, superior performance has been attained by our approach with comparing to the state-of-the-art dedicated methods. As a by-product, a CapS dataset is constructed by augmenting existing benchmark training set with additional image tags and captions. 

Joint Semantic Mining for Weakly Supervised RGB-D Salient Object Detection
Jingjing Li Wei (is0, Qi Bi?, Cheng Yan Miao Zhang", Yongri Piao', Huchuan Lu Li Cheng'
University of Alberta, Canada Wuhan University, China Tianjin University, China DDalian University of Technology, China Pengcheng Lab, Shenzhen, China
Abstract
Training saliency detection models with weak supervisions, e.g., image-level tags captions, is appealing it removes the costly demand of per-pixel annotations. Despite the rapid progress of RGB-D saliency detection fully-supervised setting, it however remains unexplored territory when only weak supervision signals are available. This paper is set to tackle the problem of weakly- seeakdy-uuprvised RGB-D salient object detection. The key insight in this effort is the idea of maintaining per- pixel pseudo-labels with iterative refinements by reconciling the multimodal input signals in our joint semantic mining (JSM). Considering the large variations in the raw depth map and the lack of explicit pixel-level supervisions, we propose spatial semantic modeling (SSM) to capture saliency-specifie depth cues from the raw depth and produce depth-refined pseudo-labels. Moreover, tags and captions are incorporated via a fill-inthhe-blank training in our textual semantic modeling (TSM) to estimate the confidences of competing pseudo-labels. At test time. our model in- volves only a Hight- sub-network the training pipeline, i.e., requires only an RGB image as input, thus allowing efficient inference. Extensive evaluations demonstrate the effectivvness our approach under the geakkl-suuerrisee setting. Importantly, our method could also be adapted to work in both uully-supervised and unsupervised paradigms. In each these scenarios, superior performance has been attained by our approach with comparing to the state-of of-the-ar art dedicated methods. As a by-product, a CapS dataset constructed by augmenting existing benchmark training set with additional image tags and captions. Code and datasel are available at https: //git//ttt herptthu..omooj.... com/ jwmm://Iwv000
Introduction
As fundamental computer vision task, salient object detection (SOD) aims at locating and segment-
ing visually distinctive objects in a scene. It plays an important role in a variety of downstream appli- cations including image retrieval
[I 63]. medical analysis 48 28 25], multimodal fusion 77
80
and video analysis [8 81 90 Recent progress in supervised RGB-D SOD (5 52
I0
26
89 33 has
demonstrated significant benefits of engaging depth information saliency detection from complex
scenes. The success of these fully-supervised methods, however, relies heavily on the large-scale, pre- cise, pixel-level annotations, which are often laborious and time-consuming to acquire. On the other hand, an image usually comes with additional information in its meta-data such as tags and captions from users to describe the scene context and content, which may serve as cheap weak-supervision signals. These weak supervision signals are nonetheless noisy and have mixed qualities. Similar weak-supervvsion signals have been explored in RGB-image based SOD 63 70 38), where the noisy nature of these side information is unfortunately overlooked. To further complicate the matter, the lack of explicit pixel-level supervision brings new challenge to the RGB-D SOD task: the depth values from depth maps are often noisy and sometimes inconsistent. For example, in Fig.
*means equal contributions. Wei Ji (wji3@ (uiluuabbrta.cc) is the corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

RGB Image
Depth Map
Initial Pseudo-label
Updated Label
Updated Label 3
Updated Label 7
Category:
Caption:
The masked Caption:
Cat
A black and white cat laying on couch. A black and white <MASK> is lnying gray couch.
----e
Figure 1: Illustration of sreakly-supervise RGB-D salient object detection. RGB and depth images,
as well as weak supervision signals such as image-level tags and captions are exploited. Initial pseudo-label is generated by the handcrafted methods, which is then iteratively updated by our joint semantic mining pipeline. GT is ground-truth label for reference.
similar depth values are shared by the cat and the underneath couch, making it difficult to discern the salient object from backgrounds. Without the explicit pixel-level supervision, existing cross-modal fusion strategies adopted by fully-supervised RGB-D methods 36 55
37 27 would simply fail.
These observations motivate us consider the new problem of wreakly-superiine object detection, which takes as input the RGB and depth images, as well as weak supervision signals such image-level tags and captions, as illustrated in Fig. D By removing the demand for laborious per-pixel annotations, it also brings new challenges: 1) how to address the noisy nature of the weak supervision signals; 2) how tackle the depth noise and inconsistency facilitate proper separation of foreground and background regions.
RGB-D salient
This leads us to propose the use of pseudo-labels with iterative refinements in training: the pseudo- label provides internal pixel-Ievel supervision signals, which is progressively updated by reconciling the multimodal input signals and the current information flow of the neural net, based on the previous pscudo-label. As illustrated in Fig. [ this is realized by an interaction between two core modules, namely spatial semantic modeling (SSM) and joint semantic mining (JSM): SSM is designed to capture the saliency-specifi depth semantics, to eliminate the background noises in the coarse saliency prediction, and generate a depth-refined pseudo-label. This simple yet effective module is very generic, which could be easily plugged-in in different setups, including unsupervised & fully- supervised scenarios; meanwhile, the JSM module is proposed to leverage depth semantics and weak supervision signals for attaining more reliable pseudo-labels. Specifically, a partial textual input, i.e., image-level tag and caption with its salient word being masked, is fed into dedicated textual semantic modeling or TSM to estimate the confidence scores of competing pseudo-labels, and to Intuitively, a semantically consistent pseudo-label should provide better context cues to reconstruct the salient word: while a closer guess of the masked word would indicate a better pseudo-label. The alternation between SSM and JSM modules is thus expected to give rise more trustworthy pseudo-labels. At test time, it is then sufficient to take an input RGB image and activate a light-weight network to deliver its final prediction. That is, test time input involves only an RGB image, without the need of any depth map or image-level tags and captions. This drastically simplifies the input requirement and reduces the computation burden at deployment stage. Moreover, given the setting, we adapt existing RGB-D training dataset
lack of training dataset for the to annotate additional lmage-level tags and captions, which referred to as the CapS dataset.
The main contributions of this paper are as follows. (1) A new problem of weakly-supervised RGB-D salient object detection is considered. In this regard, a CapS dataset curated by augmenting the existing RGB-D SOD training dataset with image-level tagging and captioning annotations. (2) The key ingredient of our approach involves the production of pseudo-labels with iterative refinements, realized by iterative updates between two internal modules, SSM and JSM. Empirical experiments demonstrate the effectiveness of our approach in superisssper setting. Moreover, (3) after proper adaptation of our approach in unsupervised and fully-supervised scenarios, superior performance is also observed when comparing to the respective state-of-the-arr methods. (4) Our test time inference amounts to executing fightwweigh saliency network: as illustrated in dotted line at Fig. 2 only an RGB image is used as its input, thus allows efficient and effective inference.
2 Related Work
RGB-D salient object detection 87 93 94 has been an active field of research in the past few years, where the incorporation of depth cues has been demonstrated [3] 34
82
75
[5
35
60 43 85

---

only
Saiiency Network
Superviaion
Saliency
Label Upddat
Visual Feature
Prediction (Prea))
Peeudo Label (Sup))
black
SCG
Spatial Supervision
<MASIC- laying
Semantic Modeling
Weighte Averagg
Gemeration
gray couch.
Image
Depth Map (Pauy
Masked Caption
Contdence
Updated (S
Predictoon (Spreel
Duscc
Vicual Focturo
Depth
Baskground
Noov
Suporession
Depth Sementics (Ds)
Depth-rafned (5p)
|-- Input -
Spatial Semantic Modeling
Joint Semantic Mining
Figure 2: An overview of our approach. Its training pipeline consists of a saliency prediction network, SSM (Sec.3.2) to generate depth-refined pseudo-label, a TSM (Sec. estimate the confidences of different pseudo-labels, and a JSM (Sec. (Sec.[3.4) to refine & update pseudo-label. Our testing process only involves activating a saliency network delineated in dotted lines. More details of SSM and TSM modules are illustrated in Fig. The masked salient word is 'cat'.
improve performance especially in complex scenes. Existing RGB-D methods aim design effective feature fusion strategies for learning representative cross-modal features. Typically, Chen et al. [8] employ two-stream CNNs-based models and perform fusion by adding or concatenating paired features at shallow or deep layers. Fu et al. [19] utilize a Siamese network to jointly learn RGB and depth inputs for mining useful complementary features. To promote multi-modal interactions, Li et al. (37 design a cross-modal weighting strategy to encourage comprchensive interactions between RGB and depth information. However, those methods unfortunately rely on costly pixel- level annotations, which are tedious and time-consuming to acquire. This motivates us to consider a weakly-supervise -sueevvised approach. In what follows, our focus will be mainly toward related weakly- supervised methods developed for saliency detection from RGB images, where the differences of our approach from existing methods would be clarified.
Instead of using costly pixel-level annotations, some recent efforts instead explore cheap alternatives 2 image captions 770] scribble labels 66 76).
such as image-level tags (categories) 38 63 71
46
and noisy pscudo-labod from handcrafted methods 45 72 73 77 68 47 67 49 66). The pioneering
work (63] leverage image-level tags or categories that could be augmented onto existing large-scale dataset at low-cost. The same scenario is also considered by Li et al. [38] where a composite pipeline combining graphical model with CNNs is designed. The trained network however tends to highlight the most discriminative region instead of the intended salient object out of the scene due to the sparse image-level supervisions. Image captions are examined by Zeng el [70 as supervision input; in their work image classification network and caption generation network are jointly trained to obtain pseudo-labels, which achieves descent performance. Scribble is another type of weak supervision signal, where a tiny fraction of image pixels are labeled by users as being foreground or background. Due to the annotation sparsity, object structure and details cannot be easily inferred. Zhang et al. [76) introduce gated structure-awart loss well as auxiliary edge detection network to uncover the complete object. Mcanwhile, Yu et al. [69] explore ceff-consiteency among multi-scale outputs and design local coherence loss propagate the labels unlabeled regions based image features, thus enabling the detection of objects with smooth textures. However, existing weakly supervised saliency methods are solely based on RGB image. Unlike the prevalence of superviud RGB-D SOD, it has never been considered the incorporation of depth cues in existing literature.
This leads us to address this problem in the presence of image tags and captions as weak supervision signals. Different from existing methods [38] 24 63 770 that train image classification networks or caption generation networks to delineate potential salient regions, masked salient word in captions to be reconstructed in our work by leveraging visual saliency features. This is then used to estimate
the confidence scores of pscudo-labels.
3 Methodology
3.1 The Overall Architecture
An overview of our approach illustrated in Fig. 2 consists of saliency network responsible for saliency prediction, spatial (depth) semantic modeling (SSM) to generate depth-refinet labud.-label,

---

black
black
Training Stage
Supervision
<MASKC- laying
<ATT>
on a gray couch.
on a gray couch.
Feature
Depth
Network
East
Image
10,) Death Samaniiaa
Dexc
Shore
Spee Spatial Supar raioon Cararaaion stiio
Visuel
Filendd
Transfomer-Ilke Not
Ohtppl Probabiilsos
Share
Depin
Label Update
Network.
Teasell
Faature
Image
ex
Vieual Peeture
Rerthed Label (5m)
Prediction (Pare))
Background Noise Suppression
Vieaa E'naturs
6Gg Confidence Score
Spafial Modeling
Textual Semantic Modeling
Figure 3: The detailed architecture of the proposed SSM and TSM. The upper shows their training processes, and the bottom illustrates the way of using them to perform label update.
textual (caption) semantic modeling (TSM) estimate the confidences of different pseudo-labels, and a joint semantic mining (JSM) strategy to refine & update pseudo-label. Overall our pipeline aims to gradually improve the quality of noisy pseudo-labels by jointly mining the useful spatial semantics from the depth map and textual semantics from the tags and captions, to produce more trustworthy supervision signals, which in turns results in better training of the saliency network.
Specifically, the popular encoder-decoder architecture [64] in SOD adopted in both saliency network and depth network. Initial supervision signal for the saliency network is provided by traditional handcrafted methods. The predicted saliency map, together with the raw depth map. are processed to generate the saliency-guided spatial supervision signal for the depth network. Then the predicted saliency-oriented depth semantics utilized to eliminate background noises (non-salient regions) in coarse prediction, and generate depth-refined pseudo-label. This is followed by our JSM strategy, which takes in the image-level tags and captions through TSM to estimate the confidence scores of pseudo-labels; updated pseudo-label is then formed based on the confidence-weiglee pseudo-label and current pseudo-label, which provides more trustworthy supervision signal for the saliency network. Note our test time inference involves only the black dashed portion in Fig. which takes as input only the RGB image, thus enables efficient saliency prediction.
depth-refined
3.2 Saliency-oriented Spatial (Depth) Semantic Modeling
Initial pscudo-labels generated by traditional saliency models, which often contain excessive noise. As illustrated in Fig. 3 our spatial semantic modeling (SSM) is to produce more reliable depth-refined pseudo-label, achieved by explicitly capturing saliency-specific depth semantics from the depth map to eliminate possible background noise in the coarse saliency prediction.
Concretely, during training, we first generate seliency-guidee depth mask D Dmask by multiplying the rough saliency prediction Spred with the raw depth map Dmmpp in spatial attention manner. Here, Gaussian smooth operation is applied smooth the predicted saliency area, to effectively perceive and capture more saliency areas from depth. The procedure is formulated as:
D Dmask Qmax (F Fgauss (Suuss(preed (Spred, k). Spred) & Dmap:
where F Faauss(( (., k) indicates convolution operation with Gaussian kernel and zero bias; 2 (::)) a maximum function to preserve the higher values between the smoothed and the original maps. 6 element-wise multiplication. In this paper, the size and standard deviation of kernel are learnable through the model training procedure and are initialized with values 32 and 4, respectively.
After obtaining Pmakk a depth network is trained to learn the saliency-specific depth semantics Ds. using the mean square error (MSE) loss function. The internal inspection evidences in Fig. 4 suggest that Ds (depth semantics, 6th column) is able to capture discriminative saliency cues from the raw depth map under the supervision of D Daak (5th column).
In addition, the learned DD5 can be further processed to generate the depth-refined pseudo-labels. This procedure is only employed when performing pseudo-label update. Specifically, we feed Ds into an Background Noise Suppression block to help climinate the background noises (non-salient

---

A vase is sta- nding next
6868
SC, 0.315
glass door.
A brown dog
413
297
standing the
wooden floor.
A white butt-
36G 0. 14549
SC, 0.451
erfly is flying
over the green grass.
Weak Annotations
RGB
Depth Map
Saliency Prediction
Spatial
Depth Semanties
Depth-Refined Refined
Initial
Updated Label
GT
Image
Supervision
Label
Label
Reference
Figure Step-by-step inspections of the internal processes of our approach. The GT is for reference.
regions) in the coarse prediction Spred. In this block, a reverse operation is first employed on Ds to highlight background regions by 1- Ds. This is followed by the pixel-wise subtraction suppress the non-salient negative responses in S Spred. Finally, a ReLU function and normalization procedure are adopted to suppress negative numbers and normalize the result the range of [0, 1]. This procedure of obtaining the depth-refined pseudo-label Sp could be expressed by
S min(Sa)
S
max((S) (Sa) + Sa ReRUU Aa E M1. E [1, (2)
Here Ad E [0, 1] is constant to control the degree of the subtracted background noises and avoid negatively suppressing salient regions that have relatively low saliency scores Sp. Throughout experiments, Aa is empirically set to 0.5, and to 1e-5. H and W are the height and width of the input image, respectively.
The pseudo-label refinement dynamics are visually inspected in Fig. while Fig. g presents the corresponding quantitative error analysis over iterations. Empirical evidence suggests as the training proceeds, quality of the pseudo-label is significantly improved. Moreover, at this stage, we can directly utilize the depth-refined label Sp to update pseudo-labels (as in Fig. (b)). This can be referred unsupervised RGB-D SOD since weak labels are used.
3.3 Saliency-oriented Textual (Caption) Semantic Modeling
Previously, the mainstream use of weak labels is to train a classification network or a caption generation network, where the by-product attention maps Class Activation Maps [92 leveraged to determine the potential salient regions 770 63 It is very different our textual semantic modeling (TSM), where the main focus is to leverage side information (i.e., image-level tags and captions) to facilitate the production of reliable training signals. Inspired by the recent success of masked language models [14), captions with missing keywords are used as input, with the expectation of the complete text being reconstructed as output. In the proposed TSM, innovatively taking as input partial text with salient word being masked, well the saliency-filtered visual features, our TSM is to output the reconstructed text in a fill-in-tee-e thanklank manner and to estimate the confidence scores of competing pseudo- labels. The intuition is, a semantically matching pseudo-labe could provide better context cues to reconstruct the masked salient word; meanwhile. closer guess of the masked word would indicate better pseudo-label.
Formally, for each training data, the weak labels contain caption description e= {c:))En image-level category k (1. K), and the position X' of the salient word (object) in the caption, where n. is the word number of the caption. Let E d be the word embedding of the i-th word in the caption, K the total number of salient categories, and X' an integral number. As shown in Fig. 3 during training, the input is masked version of caption è paxn, where the salient word ex caption e is masked with special symbol. In order to reconstruct the masked salient word, we filter the visual feature from the saliency network by multiplying with the learned saliency cues Spred- We then obtain the saliency-filtere visual feature, and transform it to feature vector f Rd for subsequent cross cross-modal modal fusion using a pooling operation and a fully-connected (FC) layer.
The center component of the TSM module is transformer-like network. Based on original Trans- former 622, we add multi-modal fusion sub-layer into network. Three parallel operations are used to promote sufficient moosl feature interactions: element-wir wise multiplication, clement-

---

addition, and concatenation followed by FC. The three outputs are concatenated and followed by FC change the feature dimension. Note the modsl fusion operation is performed word-wise. Through the textual network, we can obtain the energy vector ex E R of the masked salient word, which computed over all categories by a fully-connected layer and softmax function. For each training sample, the training objective loss for the textual network is -log(ex [A]), where ex [k] is the output probability of salient category k. Detailed structure for our TSM can be accessed in the supplementary material. Once trained, the TSM module is then used to estimate the confidence scores of pseudo-labels as described in Sec. 3.4
3,4 Joint Semantic Mining for Label Updating
The label updating operation using joint semantic mining is iteratively conducted every T epochs (as training round) during training. i.e., the granularity of label updating. The choices are explored in the ablations of Sec. 4.44 where т = 5 is shown to work best empirically. In terms of training, the saliency network, the SSM module, and the TSM module are independently trained.
At the end of each training round, our JSM strategy performs the label updating operation to generate an up-to-date pseudo-label for each training image. Now define the pseudo-label for the saliency network in current iteration as Smap. As shown in the label update phase of Fig. 3 the SSM module is engaged to generate the depth-refined pseudo-label Sp; It is passed to the TSM module, where the Shap and Sp are taken as saliency attention maps to filter visual features, respectively. The TSM module infers the energy vectors e'x and ex for St map and Sp, respectively. Thus their confidence scores SC, and SC.D can be calculated by (taking SC; as an example):
(+)) SC, exp(e> + explee
The updated label is the weighted average of map and Sp: Smap SC, x Smap + SCD x Sp. A cully-coneected conditional random field 23 is applied to generate the final updated label which could provide more trustworthy supervision signal train the saliency network.
3.5 The CapS Dataset
To train the weakly-supervisee RGB-D SOD network, we relabel two widely-used RGB-D saliency datasets, NJUD [2] and NLPR 53 which contain total of 2, 185 training images. We provide various image-level weak annotations: categories, captions, and position of the salient word in each of the captions. The annotation process is conducted semi-automatically;: the NeuralTalk2 [30] toolkit is utilized to automatically generate image captions, which are then manually checked, with unreasonable cases corrected. On average, there are 8.9 words in each caption. This is followed by categorically tagging each of the images, each corresponds to a salient category; this is different from the traditional image classification dataset ImageNet (32). We summarize the 100 categories in the CapS. The positions of the salient words are first automatically identified through localizing the image categories in the captions which is subsequently followed by manual identification. Detailed statistics and examples of our in-house CapS dataset are relegated to the supplementary material.
4 Experiments
4.1 Datasets and Evaluation Metrics
Empirical evaluations are conducted over four large-scale RGB-D SOD benchmark datasets, including
NJUD 29 with 1,985 RGB-D paired images, NLPR 53 with 1,000 samples, STERE (50) with 1,000 stereoscopic images, and DUTL---eep
55 with 1,200 RGB-D data. In train vs. test splits
of the datasets, the popular setup of [18 19 74 is followed for fair comparison. Training set consists of 1,485 samples from NJUD and 700 samples from NLPR. The remaining images in these datasets and other public test sets are reserved for testing purposes throughout the experiments, Four widely-used metrics are adopted for quantitative evaluation: they are E-measure (Ee) [16] F-measure
(F3) II weighted F-measure (FT") 44) and mean absolute error (MAE or M)
4.2 Implementation Details and Setups
The code is implemented in Pytorch toolbox on a PC with a single Tesla P40 GPU. We use decoder
part (64] with ResNet-50 [2) pre-trained on ImageNet as backbone, for both saliency network and
depth network. For each word in the caption, we extract word embedding with dimension using the pretrained Glove (54) word2vec network. The maximum caption length is set to 20. For
300

---

Table 1: Quantitative comparison with weakly-superviset and unsupervised saliency models. Note RGB-based methods are specifically marked by &, Ds and Tx represent the spatial semantic modeling and textual semantic modeling, respectively, 'Un' means unsupervised learning. 'CIs' is SOD with class label. 'Cap' represents using
weak supervision signals, with both class label and image caption.
DUTLF- 155
STERE
NJUD
NLPR
Sup.
Ag
Fe
F'a
Eg
F2
Fa
JM
Fgg
F2
Fa
M
Eg
F'a
M
RBD
98
.33
.497
.619
.222
.330
.443
.610
664
.387
.556
.256
.765
.388
.590
.211
MST
61
Un
.678
.254
.401
.279
.681
.312
.447
.269
.670
.291
.436
.281
.762
.257
.491
.199
BSCA-
57
Un
.808
.479
.682
-181
8003
.497
.676
.179
.756
.446
.623
.216
.745
.376
.554
.178
DSR 10
Un
.797
.78
.440
164
785
.486
.645
.665
739
.36
.594
.196
.757
451
.545
.120
ACSD
Un
.250
.210
188
.68
793
.455
.61
200
.700
.488
.696
.188
.751
.327
.547
.171
DES
Un
.733
386
6668
280
673
383
.592
.977
421
241
1665
448
.335
.259
.583
.301
53
.003
.119
GP 59
Un
785
.311
.710
182
730
323
(666
.204
813
.347
670
.14
CDB (40
Un
808
.436
.713
.66
.722
408
.650
.200
.810
.888
.188
.108
20
.730
.339
.474
.96
825
.546
.447
.143
780
.735
.164
.833
.578
.701
.855
DCMC 12
Un
.712
.250
.006
.243
832
.529
.4.3
148
.796
506
.715
.167
.684
.665
.328
.196
D0
.464
.492
.844
089
CDCP 977
Un
.794
.530
.633
.159
.977
.596
.666
.499
.251
.222
.618
.181
785
.522
.59
.144
Ours (Pa)
Un
.845
.629
.741
.116
.838
.654
-738
.111
.781
.576
.689
.147
.87
.638
.698
.073
WSS 63
Cls
843
634
.143
125
831
664
7732
.15
760
.66
.93
.149
856
651
.212
007
MSW1
Cap
6678
.789
105
675
.760
704
860
.736
071
Ours (13g (5)
Cap
.870
.698
.797
.093
.852
.688
.778
.095
788
.599
.77
.133
888
.692
.770
.60
Image
Depth GT
Ours
MSW
CDCP
LHM
DCMC DSR
BSCA
Figure
Visual comparison of wcakly-supervised and unsupervised saliency models.
the transformer-Bee like net in TSM, set the hidden state to 256, the number of layers to 3, and the number of head to 4. The model is optimized by Adam with batch size 10, and the learning rate is the standard BCE loss train the saliency network. Each
set to 1 x 10-4 During training, image uniformly resized to 352 x 352 and is performed by randomly rotating and cropping to avoid potential overfitting. Our network is trained in end-to-end manner and converges around 50 epochs. Initial pseudo labels are generated by the handcrafted method 997, which is freely available for annotations.
4.3 Model Performance
We quantitatively evaluate the performance of our approach in Table T with visual results shown in Fig. [ Since our approach is the first work for weakly-supervised RGB-D saliency detection, show the results of two state-of-the-are RGB-based
weakly-supervisee methods (WSS [63] and MSW (70) for reference. To make a fair comparison, we fine-tune them on the same training set using their published code with default setups. These results show the effectiveness of our proposed method. Furthermore, the SSM and TSM in our joint semantic mining framework do not introduce any additional inference cost since they only participate the training procedure to provide more reliable supervisory signals for saliency network. This design leads to light-weight network that works both efficiently and effectively. As shown in Table our method runs fastest among RGB-D methods, and also on par with the most efficient RGB-basces methods.
Table 2: Inference time of diffcrent unsuporvised and wcakly-supervised saliency models. The RGB-based methods specifically marked by" +
HM
DES
GP
CIPPP
DCMC
SR
ACSD 29
CDB
HSCAL
RHD
MSP
DSRI
MSWI
Ours
[39]
70
Interence Time (5)
211
2.79
12.8
5.7
0.118
0.6
2.655
0.1893
0.000
0.3738
0.0267
0.00%6
4.4 Empirical Analysis of Our Pipeline
Here we focus on evaluating the contribution of each component our pipeline, and examining the internal performance of the pscudo-labels at different stages in training.

---

e/E++1 Qument Pseudo Labeli Updated Pserodo
Sup. Siperviiion
Llement Addioon
Weeghtdd Averae
Pseudo Label
Pseudo Labd
Depth
Pseudo Labed
Deptt-reflned Label
Pseudo Labed
Depth refined Lubel
Label
Sup.
Sedf-update
Sup.
Sup.
Sup.
Spotial
Semantie Madaling
Prodiction
Parudo Label Updiate
Prediction
Prediction
Prediction
Depth Pecorlo Label Updatte
Depth
Depth Porrdo Label Updinte
Saliency Netwock
Network
Netwookk
Psendo Label Updasto
RGB
RGB
(b)
RGB
(c)
RGB
(d)
Figure 6: Diagrams of various label updating strategies used in our ablation study: (a) self-updating strategy,
(b) SSM updating strategy, (c) historical moving average with equal weights, and finally (d) our JSM strategy.
Table 3: Ablation study of our pipeline. 1 (J) denote performance gains (relative to backbone).
DUTLF-Depth [55]
STERE (50)
NJUD
[29
NLPR (53)
Model Setups
Fj
Fa
F3
Fa
.4
Fa
M
M
Backhone tminned on Pisado Labels (L.. (8)
.312
644
167
.55
666
.58
.100
627
186
479
.570
.126
'8' tained on Pscudo I Labels with CRF
368
660
140
601
684
.33
546
642
165
5555
.624
4994
with Self-updating Statogy
616
.077
643
708
675
.144
.607
.077
629
.41
.116
654
.338
.11
.76
.699
.147
.638
.698
.033
15' with Sputial Semantic Modeling
123%
115%
131%
118%
111%
130%
113%
110%
131%
133%
122%
.42%
with SSM Historical Moving Avemge
644
.750
674
753
588
498
.41
.722
068
.96
.797
.093
.688
.778
.955
.599
.717
.133
.692
.770
.660
'8' with Joint Semantic Mining (iee, Ours)
136%
12493
1248
117%8
1.40%8
117%
1149
1289
1449%
135%
.52%
Ablation analysis. We present in Table 3 ablation results of our pipeline on four benchmarks. To start with, we consider the backbone as the saliency network trained with initial pseudo labels. As our proposed SSM and TSM are gradually incorporated into the backbone to generate the depth-refined pseudo-labed and estimate their confidence scores for label updating, noticeable performance gains are consistently achieved in all datasets. The SSM significantly reduces the MAE metric and increases the F-measure score by 31% % and 14.5% on average in four datasets, respectively. The TSM further boosts the saliency detection performance to higher level where significant amount of performance gains with 41% % and 22.5% are finally achieved on MAE and F-measure metrics.
To further demonstrate the effectiveness of our SSM in exploiting the depth semantics to refine pseudo-labels, we retrain the saliency network using the self- uelf-upda strategy as in Fig. 66 strategy, the saliency prediction with CRF are directly utilized to update pseudo-labels, at the end of each training round. As shown 3 row vs. 4th row), when excluding saliency-oriented depth semantics captured by the SSM, the performance of model degrades greatly. This indicates our SSM can effectively suppress background noise and providing reliable training labels. Furthermore, replace TSM with a hcuristic historical moving average strategy where the pseudo-labels are assigned with equal weights as in Fig. 6 (c). Table shows achieves better performance than the backbone using SSM due to the consideration of historical information, but it is consistently inferior compared to our pipeline with the TSM module. These results suggest that our TSM can better estimate the confidence quality of pseudo-labels and generate the trustworthy supervision signals. Meanwhile, we also provide the internal inspections of our approach in Fig. 4 in terms of the generation of pseudo-label and their corresponding confidence scores, for better understanding.
F (a). In this
In addition, we further discuss the effect of dif- ferent update intervals in our pipeline when per- forming label updating. As listed in Table. [4] the larger or smaller update interval leads to inferior performance due to the insufficient or excessive learning of models.
Table 4: Parameter analysis of the update interval
(epach) in the JSM.
STERE (50
NJUD 22
NLPR [33
Interval (r)
Fg
F6.
F6
.769
.098
.703
.142
.766
.063
=
.778
.095
.717
.133
.770
.060
7 10
.758
.101
.695
.138
.763
.065
Analysis of pseudo-labels. We analyze the evolutions of pseudo-labels in our training process. Here the quality of pseudo-labels is measured using the ground-trute labels of the training set (for evaluation only). Visual evidences in Fig. 7 show that the quality of pseudo-labels is gradually improved as our JSM is performed. We can see that the initial pseudo-labels unfortunately tend to miss important salient parts as well as fine-grained details. By adopting our proposed joint semantic mining for label updating. the missing parts could be gradually retrieved, with the object silhouette

---

617
Spuiia Sensanti Modeling Label Uplating
214
Moan Absolite
0.135
0.130
0.126
0.127
0122
0.127
40-
0.124
0.118
Image
Ininil
Updazed
Updaned L
Updated
GT
Label
Label
Label
Epoch Number
Figure 8: Error reduction plot of the updated pscudo- labels over iterations.
Figure 7:
Visualization of updated labels.
Table 5: Ablation analysis of the supervised variant approach, where the human annotations (i.e., ground truths) are used to train SOD models.
CMWN
ATSA
DANet
Albele
UCNa
STERE [50]
NJUD 29
NLPR [33]
Semups (fally)
P3'
Fg
F3"
34
Pg
Image
Depth
Ground Truth Ours (Pully)
HDPNa
Backbone die.. 'B')
838
869
1945
864
871
046
865
863
028
Figure 9: Visual comparison of fully supervised RGB-D SOD methods.
18 SSM (Ouns)
.76
.866
009
885
906
033
892
905
022
also being refined. The final pseudo-label is closest to the true label, which could provide more reliable guiding signal for training saliency network. Moreover, we present Fig. (8) the error reduction curves of the updated pseudo-labels with our full pipeline (blue line) and SSM only (red line), respectively. can be seen that only SSM is able to improve the quality of pseudo-labela by exploiting the useful depth semantics to refine pseudo-labels. Our JSM further boosts the performance by leveraging the textual semantics to integrate reliable pseudo-labels.
4.5
Generalization Analysis
Adapting to unsupervised setting. Our approach be adapted unsupervised setting by using the architecture illustrated in (b), i.e., SSM, where only the depth semantics are mined to refine pseudo-labels without weak labels. The quantitative results in Table show the effectiveness of our SSM in unsupervised setting.
Adapting to fully-supervised setting. Apart from the adaption to unsupervised setting, variant of our approach can also be applied fully-supervised RGB-D SOD scenario. This is achieved by modifying the generation of saliency-guided depth mask as Dmask = ScT % Dmop in Eq. I with SGr being the ground-truu truth annotations. The saliency network and depth network are trained by ScT and maskk respectively. Then the background noise suppression block in SSM module is applied to obtain the final saliency. As ablated Table 5 our fully-supervised variant achieves consistent performance improvement compared to the backbone trained Ser. In addition, results compare favorably with those of 21 fully-supervised RGB-D saliency models, as quantitatively shown in Table 6 and qualitatively illustrated in Fig. 9 Notice that, different from existing fully-supervised RGB-D SOD methods in designing the complicated cross-modal feature interaction strategy, our method directly exploits the learned depth semantics promote saliency accuracy, which is simple yet effective and brand new for this field.
Failure Cases
Due to the sparsity weak annotations, the network is usually difficult to identify the fine-grained object bound- aries. As depicted in Fig. I0, although these models can effectively detect the salient objects, tine-grained details are still missing. A doable solution is to introduce auxiliary edge constraint during training. For example, the edge de- tection loss can be employed on the low-level features of model, which could force the model produce better
W
Image
Ground Truth
Ours
MSW
features highlighting the object details 776 41). The edge maps can be generated by classical Canny operator an unsupervised manner.
4 in
Figure 10: Failure cases existing
weakly-supervised saliency methods.

---

Table 6: Quantitative comparison of existing fully-supervised RGB-D SOD methods. Notice that, in fully- supervised scenario, when evaluating the newly released DUTLF-Depth dataset, the specific setup used by [55] is adopted to make a fair comparison, Le., using total of 2,985 training samples that contain 1,485 from NJUD, 700 from NLPR and 800 from DUTLF- DUThF-Depth.
DUTLF-Depti
55
STERE
50)
NJUD
29
NLPR
[33
Method
Fig
Fg
Fg
.M
Fi
F2
F5
Fig
F2
M
CTMP TMSP
21
.884
.690
.792
.097
.841
.747
.771
.086
.864
.732
.788
.085
.669
.691
.723
.056
DF T0P 58
.442
.542
.748
.15
.691
.596
.742
.141
8188
.552
.744
.151
888
.524
.682
.099
PCA CYPR78
.58
.760
.100
.801
.826
064
8966
.844
.059
.772
044
TANer
866
.712
.779
.093
8933
804
835
060
893
.822
.844
.061
.916
.789
.75
.411
PDNet RCME
95
.861
.650
.757
.112
.880
.799
.813
.071
.90
.798
.832
.062
.876
.659
.740
.664
MMC| PM
855
636
7553
113
873
.55
.829
068
878
.749
813
.079
.811
688
.729
055
CPFP UVPR'
86
.814
.644
.736
099
9122
808
.830
.051
895
.837
.850
.053
.924
820
822
.03
DMRA JOPe
-927
8588
.883
048
841
.876
044
.008
.872
.942
.845
031
SSF 83
.946
894
.914
.034
.921
.850
867
.046
.913
.871
.886
.043
.949
.874
.875
.026
A2dele CVre3
56
-924
864
.890
043
.915
855
.874
.044
897
.851
.874
.051
.45
.867
.878
.088
ILDDCECWWa 19
-931
.863
8833
.043
.857
.869
.944
.882
.878
.02
S2MA Chreray
42
-921
861
.866
044
.907
825
.855
.051
.388
.822
.853
.000
L'CNr C1PM2
.903
.821
.856
.06
.922
.867
.885
.039
.953
.878
.05
PGAR ACCY220
9
9944
.889
.914
.035
.856
.880
.44
.915
.871
.893
.042
.955
.881
.885
.024
DaNer
.847
.668
.756
097
.220
.845
855
046
913
866
8633
.047
.943
.854
.857
.000
ACEV 37
.916
.666
.056
.847
-910
8888
.940
.856
.02
BBSNer
[B]
.833
.663
.774
.120
.955
.588
885
041
.924
884
902
.035
.952
.879
.882
.0.3
DANer ACCV229
[]]
.955
.847
.884
.047
.914
.330
.858
.047
.949
.858
.871
.028
FRDT 4030012
[44
941
.878
.902
.099
.955
8588
.872
042
-917
.622
.879
.048
.946
.863
.868
.029
ATSA BCCV22
78
.947
9001
.918
.032
.919
.666
.874
.040
.21
.883
.893
.040
.945
.867
.876
.028
HDFNer 20
934
040
863
040
9155
.933
.48
.869
027
Ours (Fully Sup.J
.449
908
933
4030
.29
.76
896
.039
926
885
.906
.058
.959
.892
.905
.022
Conclusion
To tackle the new problem of rreakly-supervin RGB-D salient object detection, we propose in this paper an end-to-end approach based on iterative updates of the internal pseudo-labed This allows us to leverage depth information in eliminating non-salient background noises and generating reliable depth-refined pscudo-labels. Textual semantics is incorporated in the all-in-the-bler fashion, which is used estimate the confidence scores of pseudo-labels. Extensive experiments demonstrate the effectiveness and efficiency of our approach. In addition, our method is very generic and can be easily adapted to fully-supervised and unsupervised paradigms. In these scenarios, our variants also obtain superior performance over the existing state-of-the-art dedicated methods.
Acknowledgement. This research was supported by the University of Alberta Start-up Grant, UAHJIC Grants, and NSERC Discovery Grants (No. DBPIN-2011--45755 The authors are grateful to the anonymous reviewers for their valuable suggestions in improving the quality of the paper.
References
[1] Radhakrishna Achanta, Sheila Hemami, Francisco Estrada, and Sabine Susstrunk. Frequency-tune salient region detection. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pages 1597-1604, 2009.
[2] Qi Bi, Shuang Yu, Wei Ji, Cheng Bian, Lijun Gong, Hanruo Liu, Kai Ma, and Yefeng Zheng. Local-global dual perception based deep multiple instance learning for retinal disease classification. In MICCAL, pages 55-64. Springer, 2021.
[3] Ali Borji, Ming Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: A benchmark. IEEE Transactions on Image Processing, 24(12):57065572) 2015.
[4] John Canny, A computational approach to edge detection. JEEE Transactions on Pattern Analysis and Machine Intelligence, 866)66796698, 1986.
[5] Chenglizhao Chen, Jipeng Wei, Chong Peng, and Hong Qin. Depth-quality salient object detection. IEEE Transactions on Image Prucessing, 30:2350 2363,, 2021.
[6] Hao Chen and Youfu Li. Progressively complementarity-awee fusion network for RGB-D salient object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3051-3060, 2018
[7] Hao Chen and Youfu Li. Three-strean attention- aware network for RGB-D -GB-Dsae salient object detection. IEEE Transactions Image Processing, 28(6)22825 2213552283,511 2019.

---

[8] Hao Chen, Youfu Li, and Dan Su. Multi-modal fusion network with multi-scale multi- and cross- interactions for RGB-D salient object detection. Pattern Recognition, 86:376-385, 2019.
[9] Shuhan Chen and Yun Fu. Progressively guided altemate refinement network for RGB-D salient object detection. In Proceedings of the Earopean Conference on Computer Vision, 2020.
[10] Zuyao Chen, Rummin Cong, Qianqian Xu, and Qingming Huang. DPANet: attention network for RGB-D salient object detection. IEEE Transactions on Image Processing, 2020.
Depth potentiality gated
[11] Yupeng Cheng. Huazhu Fu, Xingxing Wei, Jiangjian Xiao, and Xiaochun Cao. Depth enhanced salieney detection method. Proceedings of International Conference on Internet Multimedia Computing and Service, pages 23-27, 2014.
[12] Runmin Cong, Jianjun Lei, Changqing Zhang, Qingming Huang, Xiaochun Cao, and Chunping Hou. Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion. IEEE Signal Processing Letters, 23(6):819 823, 2016.
[13] Karthik Desingh, K Madhava Krishna, Deepu Rajan, and CV Jawahar. Depth really matters: Improving visual salient region detection with depth. In British Machine Vision Conference, 2013.
[14] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina N. Toutanova. BERT: Pre- tre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171 4186, 2018.
[15] Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Strueture-measure: A new way to evaluate foreground maps. In Proceedings of the IEEE International Conference on Compuder Vision, pages 4548-4557, 2017.
[16] Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, and Ali Borji. Enhanced- allgnment measure for binary foreground map evaluation. In Proceedings of the 27th International Joins Conference on Artificial Intelligence, pages 698-704, 2018
[17] Deng-Ping Fan, Zheng Lin, Zhao Zhang, Menglong Zhu, and Ming-Ming Cheng. Rethinking RGB-D salient object detection: Models, data sets, and large-scale benchmarks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[18] Deng-Ping Fan, Yingjic Zhai, Ali Borji, Jufeng Yang, and Ling Shao. BBS-Net: RGB-D salient object detection with a bifurcated backbone strategy network. In Proceedings of the European Conference on Computer Vision, 2020.
[19] Keren Fu, Deng-Ping Fan, Ge-Peng Ji, and Qijun Zhao. JL-DCF: Joint learning and fusion framework for RGB-D salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3052-3062, 2020.
[20] Jingfan Guo, Tongwei Ren, and Jia Bei. Salient object detection for RGB-D image via saliency evolution. In IEEE International Conference on Multimedia and Expo, pages 1-6, 2016.
[21] Junwei Han, Hao Chen, Nian Liu, Chenggang Yan, and Xuelong Li. CNNs based RGB-D saliency detection via cross-view transfer and multiview fusion. IEFE Transactions on Cybermetics, 48(1 1):17113318,, 2017.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
(23] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip HS Torr. Deeply supervised salient object detection with short connections. In Proceedings of the IEFE Conference on Computer Vision and Pottern Recognition, pages 3203-3212, 2017.
[24] Kuang-Jui Hsu, Yen-Yu Lin, and Yung-Yu Chuang. Weakly supervised salient object detection by learning classifier-driven map generator. IEEE Transactions on Image Processing, 28(11):5435 5449, 2019.
[25] Wei Ji, Wenting Chen, Shuang Yu, Kai Ma, Li Cheng, Linlin Shen, and Yefeng Zheng. Uncertainty quantification for medical image segmentation using dynamic label factor allocation among multiple raters. In MICCAI on QUBIQ Workshop. 2020.
[26] Wei Ji, Jingjing Li, Shuang Yu. Miao Zhang, Yongri Piao, Shunyu Yao, Qi Bi. Kai Ma. Yefeng Zheng, Huchuan Lu, and Li Cheng. Calibrated RGB-D D salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9471-9481. June 2021.

---

[27] Wei Ji. Jingjing Li. Miao Zhang, Yongri Piao, and Huchuan Lu. Accurate RGB-D salient object detection via collaborative learning. In Proceedings of the European Conference Computer Vision, pages 52-69, 2020.
[28] Wei Ji, Shuang Yu, Junde Wu, Kai Ma. Cheng Bian, Qi Bi, Jingjing Li. Hanruo Liu, Li Cheng, and Yefeng Zheng. Learning calibrated medical image segmentation via multi-rater agreement modelingg In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 12341-12351. June 2021.
[29] Ran Ju, Ling Ge. Wenjing Geng. Tongwei Ren, and Gangshan Wu. Depth saliency based anisotropic center-surround difference. In IEEE International Conference on Image Processing, pages 1119-1119. 2014.
[30] Andrej Karpathy and Li Fei-Fei. Deep visual-semantee alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2015.
[31] ByoungChul Ko, Soo Yeong Kwak, and Hyeran Byun. SVM-based salient region(s) extraction method for image retrieval. Proceedings of the 17th International Conference on Pattern Recognition, volume pages 977-980, 2004.
[32] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, pages 1097-1105, 2012.
[33] Chongyi Li, Seed Anwar, Junhui Hou, Runmin Cong, Chunle Guo, and Wenqi Ren. Underwater image enhancement via medium cransmission-guide multi-color space embedding. IEEE Transactions on Image Processing, 2021.
[34] Chongyi Li, Runmin Cong. Junhui Hou, Sanyi Zhang, Yue Qian, and Sam Kwong. Nested network with two-stream pyramid for salient object detection in optical remote sensing images. IEFF Transactions on Geoscience and Remote Sensing, 57(1 1115699916-9 2019.
[35] Chongyi Li, Rummin Cong. Sam Kwong, Junhui Hou, Huazhu Fu, Guopu Zhu, Dingwen Zhang, and Qingming Huang. ASTF-Net: Attention steered interweave fusion network for RGB-D salient object detection. IEEE Transactions on Cyhernetics, 1P( 2020.
[36] Chongyi Li, Runmin Cong. Yongri Piao, Qianqian Xu, and Chen Change Loy. RGB-D D salient object detection with cross modality modulation and selection. In Proceedings of the European Conference on Computer Vision, 2020.
[37] Gongyang Li, Zhi Liu, Linwei Ye, Yang Wang, and Haibin Ling, Cross-modal weighting network for RGB - salient object detection. In Proceedings of the European Conference on Computer Vision, 2020.
[38] Guanbin Li, Yuan Xie, and Liang Lin. Weakly supervised salient object detection using image labels. In The AAAI Conference on Artificial Intelligence, 2018
[39] Xiaohui Li. Huchuan Lu, Lihe Zhang, Xiang Ruan, and Ming-Hsuan Hsuan Yang. Saliency detection via dense and sparse reconstruction. In Proceedings of the IEFE International Conference on Computer Vision, pages 2976-2983, 2013.
[40] Fangfang Liang. Lijuan Duan, Wei Ma. Yuanhua Qiao, Zhi Cai, and Laiyun Qing. Stereoscopic saliency model using contrast and
prior. Neurocomputing, 555:2227-2238 2018
[41] Jiang-Jian Jiang Liu, Qibin Hou, Mlng-Ming Cheng, Jiashi Feng, and Jianmin Jiang. A simple pooling-based based design for real- time salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3917 3926, 2019.
[42] Nian Liu, Ni Zhang, and Junwei Han. Learning selective setf-mutual attention for RGB-D saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 13756-13765, 2020.
[43] Yunqiu Lv, Jing Zhang, Yuchao Dai, Aixuan Li, Bowen Liu, Nick Bames, and Beng-Ping Fan. Simultanc- ously localize, segment and rank the camouflaged objects. CVPR, 2021.
[44] Ran Margolin, Lihi Zelnik-Manor and Ayellet Tal. How to evaluate foreground maps? In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2014.
[45] Tam Nguyen, Maximilian Dax, Chaithanya Kumar Mummadi, Nhung Ngo, Thi Hoai Phuong Nguyen, Zhongyu Lou, and Thomas Brox. DeepUSPS: Deep robust unsupervised saliency prediction via self- supervision. In Advances in Neural Information Processing Systems, pages 204-20 214, 2019.

---

[46] Munan Ning, Cheng Bian, Donghuan Lu, Hong- Yu Zhou, Shuang Yu. Chenglang Yuan, Yang Guo, Yaohua Wang, Kai Ma, and Yefeng Zheng. A macro-micro weakly-supervised framework for as-oct tissue segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 725-734, Springer, 2020.
[47] Munan Ning, Cheng Bian, Dong Wei, Shuang Yu, Chenglang Yuan, Yaohua Wang. Yang Guo, Kai Ma, and Yefeng Zheng. A new bidirectional unsupervised domain adaptation segmentation framework. In International Conference on Information Processing in Medical Imaging, pages 492-503. Springer, 2021.
[48] Munan Ning, Cheng Bian, Chenglang Yuan, Kai Ma, and Yefeng Zheng. Ensembled resunet for anatomical brain barriers segmentation. Segmentation, Classification, and Registration of Multi-modality Medical Imaging Data, 12587:27, 2021.
[49] Munan Ning, Donghuan Lu, Dong Wei, Cheng Bian, Chenglang Yuan, Shuang Yu, Kai Ma, and Yefeng Zheng. Multi-anchor active domain adaptation for semantic segmentation. arXiv preprint arXiv:2/08 208.000020 2021.
[50] Yuzhen Niu, Yujie Geng, Xueqing Li, and Feng Liu. Leveraging stereopsis for saliency analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 454 454-461,22 2012.
[51] Youwci Pang, Lihe Zhang, Xiaoqi Zhao, and Huchuan Lu. Hicrarchical dynamic filtering network for RGB-D salient object detection. In Proceedings of the European Conference on Computer Vision 2020.
[52] Youwei Pang, Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu. Multi-scale interactive network for salient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9413-9422, 2020.
[53] Houwen Peng. Bing Li, Weihua Xiong, Weiming Hu, and Rongrong Ji. RGBD salient object detection: benchmark and algorithms. In Proceedings of the European Conference Computer Vision, pages 92-109, 2014.
[54] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. pages 1532-1543, 2014.
20144
[55] Yongri Piao, Wei Ji, Jingjing Li, Miao Zhang, and Huchuan Lu. Depth-induced multi-scale recurrent attention network for saliency detection. In Procedingss of the IEFE International Conference on Computer Vision, pages 7254-7263, 2019.
[56] Yongri Piao, Zhengkun Rong, Miao Zhang, Weisong Ren, and Huchuan Lu, A2dele: Adaptive and attentive depth distiller for efficient RGB-D - salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9060 9069, 2020.
[57] Yao Qin, Huchuan Lu, Yiqun Xu, and He Wang. Saliency detection via cellular automata. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 110-119, 2015.
[58) Liangqiong Qu, Shengfeng He, Jiawei Zhang, Jiandong Tian, Yandong Tang, and Qingxiong Yang. RGBD salient object detection via deep fusion. IEEE Transactions on Image Processing. 26(5):2274-2285, 2017.
[59] Jianqiang Ren. Xiaojin Gong. Lu Yu. Wenhui Zhou, and Michael Ying Yang. Exploiting global priors for RGB-D saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 25-33 25-32, 2015.
[60] Riku Shigematsu, David Feng, Shaodi You, and Nick Barnes. Learning RGB-D salient object detection us- ing background enclosure, depth contrast, and top-down features. In P Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 2757, 2017.
[61] Wei-Chih Tu, Shengfeng He, Qingxiong Yang, and Shao-Yi Yi Chien. Real-time salient object detection with minimum spanning tree. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2334-2342, 2016.
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008, 2017.
[63] Lijun Wang, Huchuan Lu, Yifan Wang. Mengyang Feng. Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with level supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 136-145, 2017.

---

[64] Zhe Wu. Li Su, and Qingming Huang. Cascaded partial decoder for fast and accurate salient object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3907-3916, 2019.
[65] Cheng Yan, Guansong Pang. Lei Wang, Jile Jiao, Xuetao Feng, Chunhua Shen, and Jingjing Li. Bv-person: A large-scale dataset for bird-view person re-identification. Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 10943-10952, October 2021.
[66] Qingsong Yao, Zecheng Hc, Hu Han, and S Kevin Zhou. Miss the point: Targeted adversarial attack on multiple landmark detection. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 192-702. Springer, 2020.
[67] Qingsong Yao, Quan Quan, Li Xiao, and S Kevin Zhou. One-shot medical landmark detection. arXiv preprint arXiv:2 045277 2021.
[68] Qingsong Yao. Li Xiao, Peihang Liu, and S Kevin Zhou. Label-free segmentation of covid-19 19 lesions in lung ct. IEEE Transactions on Medical Imaging, 2021.
[69] Siyue Yu, Bingfeng Zhang, Jimin Xiao, and Eng Gee Lim. Structure-consistent weakly supervised salient object detection with local saliency coherence. In The AAAI Conference on Artificial Intelligence, 2021.
[70] Yu Zeng, Yunzhi Zhuge. Huchuan Lu, Lihe Zhang, Mingyang Qian, and Yizhou Yu. Multi Mourie weak supervision for saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 6074-6083, 2019.
[71] Dingwen Zhang, Junwei Han, Gong Cheng, and Ming-Hsuan Yang. Weakly supervised object localization and detection: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[72] Dingwen Zhang, Junwei Han, and Yu Zhang. Supervision by fusion: Towards unsupervised learning of deep salient object detector. In Proceedings of the IEFE International Conference on Computer Vision, pages 4048-4056, 2017.
[73] Dingwen Zhang, Junwei Han, Yu Zhang, and Dong Xu. Synthesizing supervision for learning deep saliency network without human annotation. IEEE Transactions on Puttern Analysis and Machine Intelligence. 42(7):1755 1769, 2019.
[74] Jing Zhang, Deng-Ping Fan, Yuchao Dai, Saced Anwar, Fatemch Sadat Salch, Tong Zhang, and Nick Barnes. UC-Net: Uncertainty inspired RGB-D saliency detection via conditional variational autoencoders. In Proceedings of the IELE Conference on Computer Vision and Puttern Recognition, pages 1592-8591, 2020.
[75] Jing Zhang, Nick Barnes Jianwen Xie, and Ping Li. Learning generative vision transformer with energy- based latent space for saliency prediction. In NeurlPS, 2021.
[76] Jing Zhang, Xin Yu, Aixuan Li. Peipei Song, Bowen Liu, and Yuchao Dai. Seakly-supervised salient object detection via scribble annotations. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 12546-12555, 2020.
[77] Jing Zhang, Tong Zhang, Yuchao Dai, Mchrtash Harandi, and Richard Hartley. Deep unsupervised saliency detection: A multiple noisy laheling perspective. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 9029-9038, 2018.
[78] Miao Zhang, Sun Xiao Fei, Jie Liu, Shuang Xu, Yongri Piao, and Huchuan Lu. Asymmetric two-stream architecture for accurate RGB-D saliency detection. In Proceedings of the European Conference on Computer Vision, 2020.
[79] Miao Zhang, Wei Ji, Yongri Piao, Jingjing Li, Yu Zhang, Shuang Xu, and Huchuan Lu. LFNet: Light field fusion network for salient object detection. IEEE Transactions on Image Processing, 29:6276-6287, 2020.
[80] Miao Zhang, Jingjing Li, Wei Yongri Piao, and Huchuan Lu. Memory- decoder for light field salient object detection. In Advances in Neural Information Processing Systems, pages 896 6066990,222 2019.
[81] Miao Zhang, Jie Liu, Yifei Wang, Yongri Piao, Shunyu Yao, Wei Ji, Jingjing Li, Huchuan Lu, and Zhongxuan Luo. Dynamic context-sensitive filtering network for video salient object detection. In Proceedings of the IEEE International Conference Computer Vision, 2021.
[82] Miao Zhang, Tingwei Liu, Yongri Piao. YuunYu Yao, and Huchuan Lu, Auto-msfined Search multi-scale fusion network for salient object detection. In ACM Multimedia Conference 2021, 2021.

---

[83] Miao Zhang, Weisong Ren, Yongri Piao, Zhengkun Rong, and Huchuan Lu. Select, supplement and focus for RGB-D saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition, pages 3472-3481, 2020.
[84] Miao Zhang. Yu Zhang, Yongri Piao, Beiqi Hu, and Huchuan Lu. Feature reintegration over differen- tial treatment: A top-down and adaptive fusion network for RGB-D salient object detection. In ACM International Conference on Multimedia, 2020.
[85] Qijian Zhang, Runmin Cong, Junhui Hou, Chongyi Li. and Yao Zhao. Coadnet: Collaborative aggregation- and-distribttin networks for co-salient object detection. NeurIPS, 2020.
[86] Jia- Jia-Xing Zhao, Yang Cao, Deng- Leng-Ping Fan, Ming-Ming Cheng, Xuan-Yi Yi Li, and Le Zhang. Contrast prior and fluid pyramid integration for RGBD salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3927-3936, 2019.
[87] Jiawei Zhao, Yifan Zhao, Jia Li, and Xiaowu Chen, Is depth really necessary for salient object detection? In ACM International Conference on Multimedia, 2020.
[88] Xiaoqi Zhao, Youwei Pang, Jiaxing Yang, Lihe Zhang, and Huchuan Lu. Multi-source fusion and automatic predictor selection for zero-shot video object segmentation. arXiv preprint arXiv:2108.05:: 05076, 2021.
[89] Xiaoqi Zhao, Youwei Pang, Lihe Zhang, Huchuan Lu, and Lei Zhang, Suppress and balance: A simple gated network for salient object detection. In Proceedings of the European Conference on Computer Vision, 2020.
[90] Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu. Automatic polyp segmentation via multi-scale subtraction network. In International Conference Medical Image Computing and Computer-Asiitted Intervention. pages 120-130. Springer, 2021.
[91] Xiaoqi Zhao, Lihe Zhang, Youwei Pang, Huchuan Lu, and Lei Zhang. A single stream network for robust and real-time RGB-D salient object detection. Proceedings of the European Conference on Computer Vision, 2020.
[92] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2921-2929, 2016.
[93] Tao Zhou, Deng-Ping Fan, Ming-Ming Ming Cheng. Jianbing Shen, and Ling Shao. RGB-D salient object detection: A survey. Computational Visual Media, pages 1-33, 2021.
[94] Tao Zhou, Huazhu Fu, Geng Chen, Yi Zhou, Deng-Ping Fan, and Ling Shao. Specificity-preseviing rgb-d saliency detection. Proceedings of the IEEE International Conference on Computer Vision, 2021.
[95] Chunbiao Zhu, Xing Cai, Kan Huang, Thomas H Li. and Ge Li. PDNet: Prior-model guided depth- enhanced network for salient object detection. In IEEE International Conference on Multimedia and Expo, pages 199-204, 2019.
[96] Chunbiao Zhu Ge Li, Xiaoqiang Guo, Wenmin Wang, and Ronggang Wang. A multilayer backpropagation saliency detection algorithm based depth mining, In International Conference Computer Analysis of Images and Patterns, pages 14-23, 2017.
[97] Chunbiao Zhu, Ge Li, Wenmin Wang, and Ronggang Wang. An innovative salient object detection using center-dark channel prior. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 1509-1515, 2017.
[98] Wangjiang Zhu, Shuang I. Liang, Yichen Wei, and Jian Sun. Saliency optimization from robust background detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2814-2821, 2014.


Reviews:
[{"review_id": "OT9FDAfdTZ1", "reviewer": "Reviewer_DUMJ", "summary": "Training saliency detection models with weak supervisions, e.g., image-level tags or captions, is appealing as it removes the costly demand of per-pixel annotations. The manuscript proposed a novel joint semantic mining for weakly supervised RGB-D salient object detection. Specifically, the proposed framework maintain per-pixel pseudo-labels with iterative refinements by reconciling the multimodal input signals. Moreover, the authors also prepared a new dataset that is constructed by augmenting existing benchmark training set with additional image tags and captions. Sufficient experiments on four benchmarks show that the proposed method surpass the state-of-the-art methods. The shortcomings lie in the following aspects: the explanation of the overall framework is not clear enough, take figure 2 as an example, there are too many internal inputs, and the data flow is chaotic. \nPlease check the following details.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "a) The main idea is reasonable and novel.\nThe manuscript proposed a novel joint semantic mining for weakly supervised RGB-D salient object detection. Specifically, the proposed framework maintain per-pixel pseudo-labels with iterative refinements by reconciling the multimodal input signals. Moreover, the authors also prepared a new dataset that is constructed by augmenting existing benchmark training set with additional image tags and captions.\nb) Extensive experiments.\nIn the experiment section, the evaluation on 4 benchmarks demonstrates that the proposed framework attains superior performance to previous models.\nc) The overall pipeline is well described and easy to follow.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "a) The experiments were extensive, but the manuscript placed the ablation experiment at the front of the Experiment section, which was not very friendly for quick understanding of the paper. I suggest the authors analyze the model performance (Sec. 4.4) at the beginning of the Experiments section.\n \nb) The manuscript is well-written, but the explanation of the overall framework is not clear enough, take figure 2 as an example, there are too many internal inputs, and the data flow is chaotic. \n \nc) I wonder how to prevent the model from going down in the wrong direction.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "CvJ1mU1SeoE", "reviewer": "Reviewer_n5it", "summary": "This article addresses salient object detection using depth maps and image captions as supervised signals. The proposed network includes a SSM for spatial semantic modelling and a TSM for cross-modal semantic mining. Additionally, a CapS dataset is built to extend existing datasets with extra weak annotations.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PROS:\n- The paper is well written and easy to follow, with clear, complete and concise explanations.\n- The paper provides a good and detailed description of the proposed network.\n- The ablation study is comprehensive.\n\nCONS:\n- My major concern goes to the design of network architecture, especially the necessity of the depth network. How about using $D_\\text{mask}$ directly for $D_s$? Please clarify the motivation to predict a depth map rather than directly using the true depth map?\n- The approach exploits a combination of multiple sources (\\ie, depth maps, captions, tags)，which leads to unfair comparisons with some leading methods that only rely on one or two types of knowledge. This make the results somewhat unconvinced.\n- L339 claims the model to be end-to-end learnable, however, the modules are \"independently trained\" (L207). I am guessing whether the network can be end-to-end trained in practise. It will be good to study the performance difference.\n- Additionally, is CRF applied during every update? How is it implemented to be end-to-end trainable?\n- How does the proposed model perform in the tailed categories? The long-tail problem seems to be obvious in CapS (as shown in Fig. 2 in the supplement)\n- It is not that obvious why does the model achieves the best results in the fully supervised setting, since the model will degrade to a simple two-branch neural network. More discussions are preferred.\n- Only RGB-D datasets are taken into account in the experiments. What's the reason for this, especially considering that the network does not rely on depth information during inference? What's the model performance on RGB only benchmarks?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. The supplement has provided some discussions of the limitations of the work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "0Poq0DW1ec", "reviewer": "Reviewer_bzEb", "summary": "In this paper authors propose to do saliency object detection with weak supervisions. To train the saliency network, pseudo labels are generated and refined step-by-step, by involving the depth map and the caption completion net. Authors relabeled NJUD and NLPR datasets with off-the-shelf image captioning model. Experiments are carried out on four SOD benchmark datasets and demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes to use a weakly-supervised manner to train saliency object detection network, which is an interesting and useful topic. It avoid extra test-time cost so the efficiency of inference is obtained.\n\nThe main comparison is made to MSW considering the framework and supervision used in the proposed method. However, this paper and MSW seem to evaluate on different testing datasets and the authors of this paper retrain the MSW model on the new datasets. It would be interesting to have a better aligned evaluation and comparison, if it is possible to reuse some of the datasets used in the previous paper.\n\nThe proposed method is also adapted to the fully-supervised settings, and it is interesting to see that the proposed method can obtain higher performance than other state-of-the-arts. More experiments and discussions are necessary to justify why the updated Eq 1 can help with the model training even if we already have the ground-truth supervision.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have adequately addressed the limitations and potential negative societal impact of their work, or there are no obvious issues regarding that direction.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper finally receives mixed reviews with slight preference to being accepted. We thoroughly check the review comments of Reviewer n5it (the only reviewer who has ‘reject’ opinion) and the authors’ responses to them in the rebuttal. Although the n5it submitted no post-rebuttal review, from our perspective, his or her comments mostly fall into technical questions (not some fundamental ones for this work), and the authors’ rebuttal seems to be mostly satisfactory to clarify the concerns. This is the ground for our recommendation, and the authors may need to reflect their own responses about the novelty and additional experiments in the final draft along with the source code. 
Decision: Accept (Poster)