Promoting Saliency From Depth: Deep Unsupervised RGB-D Saliency Detection
Authors: Wei Ji
Jingjing Li
Qi Bi
chuan guo
Jie Liu
Li Cheng
Venue: ICLR 2022 Poster (2022)
Keywords: RGB-D saliency detection
salient object detection
deep learning
unsupervised learning

Abstract: Growing interests in RGB-D salient object detection (RGB-D SOD) have been witnessed in recent years, owing partly to the popularity of depth sensors and the rapid progress of deep learning techniques. Unfortunately, existing RGB-D SOD methods typically demand large quantity of training images being thoroughly annotated at pixel-level. The laborious and time-consuming manual annotation has become a real bottleneck in various practical scenarios. On the other hand, current unsupervised RGB-D SOD methods still heavily rely on handcrafted feature representations. This inspires us to propose in this paper a deep unsupervised RGB-D saliency detection approach, which requires no manual pixel-level annotation during training. It is realized by two key ingredients in our training pipeline. First, a depth-disentangled saliency update (DSU) framework is designed to automatically produce pseudo-labels with iterative follow-up refinements, which provides more trustworthy supervision signals for training the saliency network. Second, an attentive training strategy is introduced to tackle the issue of noisy pseudo-labels, by properly re-weighting to highlight the more reliable pseudo-labels. Extensive experiments demonstrate the superior efficiency and effectiveness of our approach in tackling the challenging unsupervised RGB-D SOD scenarios. Moreover, our approach can also be adapted to work in fully-supervised situation. Empirical studies show the incorporation of our approach gives rise to notably performance improvement in existing supervised RGB-D SOD models.

Published as a conference paper at ICLR 2022
PROMOTING SALIENCY FROM DEPTH: DEEP UNSUPERVISED RGB-D SALIENCY DETECTION
Wei Ji', Jingjing Li Bi Chuan Guo', Jie Liu', Li Cheng'
University of Alberta, Canada Wuhan University, China DDalian University of Technology, China {wi13, jinglinl, eguo2, Leaeng5)@ua lbbettaa.c
ABSTRACT
Growing interests in RGB-D salient object detection (RGB-D SOD) have been witnessed in recent years, owing partly to the popularity of depth sensors and the rapid progress of deep learning techniques. Unfortunately, existing RGB-D SOD methods typically demand large quantity of training images being thoroughly an- notated at pixel-level. The laborious and consumonn manual annotation has become real bottleneck in various practical scenarios. On the other hand, current unsupervised RGB-D SOD methods still heavily rely handcrafted feature rep- resentations. This inspires us to propose in this paper a deep unsupervised RGB-I) saliency detection approach, which requires no manual pixel-level annotation dur- ing training. It is realized by two key ingredients in our training pipeline. First, a septt-diseettngleed saliency update (DSU) framework designed to automati- cally produce pseudo-labels with iterative follow-up refinements, which provides more trustworthy supervision signals for training the saliency network. Second, an attentive training strategy is introduced to tackle the issue of noisy pseudo-labels, by properly re-weighting to highlight the more reliable pseudo-labels. Extensive experiments demonstrate the superior efficiency and effectiveness of our approach in tackling the challenging unsupervised RGB-D SOD scenarios. Moreover, our approach can also be adapted to work in fully-supervised situation. Empirical studies show the incorporation of our approach gives rise to notably performance improvement in existing supervised RGB-D SOD models.
INTRODUCTION
The recent development of RGB-D salient object detection, i... RGB-D SOD, is especially fueled by the increasingly accessible 3D imaging sensors GGancola et al. 2018) and their diverse applications, including image caption generation (Xu et al.) (015)) image retrieval (Ko et al. 2004 Shao & Brady) 2006) and video analysis (Liu et al.) [2008] Wang al. 2018). to name few. Provided with multi- modality input of an RGB image and its depth map, the task of RGB-D SOD is to effectively identify and segment the most distinctive objects in scene.
The state-of the- art RGB-D SOD approaches et 2021a Ji et al. 2020b Chen et al. 2020b
Li et al. 2020b 2021b) typically entail an image-to-mask mapping pipeline that based on the powerful deep learning paradigms of e.g., VGG16 (Simonyan & Zisserman 2015) or ResNet50 (He ct al., 2016). This strategy has led to excellent performance. On the other hand, these RGB-I D SOD methods are fully supervised, thus demand significant amount of pixel-level training anno- tations. This however becomes much less appealing in practical scenarios, owing to the laborious and time-consuming process in obtaining manual annotations. It is therefore natural and desirable to contemplating unsupervised alternatives. Unfortunately. existing unsupervised RGB-D SOD meth- ods, such as global priors (Ren et al., 2015), center prior (Zhu et al. (017b), and depth contrast prior (Ju et al., 2014), rely primarily on handcrafted feature representations. This is in stark contrast to the deep representations learned by their supervised SOD counterparts, which in effect imposes severe limitations on the feature representation power that may otherwise benefit greatly from the potentially abundant unlabeled RGB-D images.
These observations motivate us to explore new problem of deep unsupervised RGB-D saliency de- tection: given an unlabeled set of RGB-D images, deep neural network trained to predict saliency without any laborious human annotations in the training stage. A relatively straightforward idea is to exploit the outputs from traditional RGB-D method as pseudo-labels, which are internally employed
"Corresponding author

---

alleviate the Published as conference paper at ICLR 2022
Ours
RGB-based method
Waighted F-meaaure (%)
80
RGBSD- (0B8D-based method
SE
CDCP
Baseline
RGB Image
Depth Map
Ground
DSR
40
GDB
BSCA
ACSD
GP
LHM
MST
DES
DCMC
6 Inference time (8)
10
12
Label
Baseline
Ours
(a) Deep unsupervised RGB-D saliency detection
(b) Accuracy and inference speed comparison
Figure (a) An illustration of deep unsupervised RGB-D saliency detection. 'Initial label' is generated by traditional method. 'Baseline' shows the saliency map generated by saliency network trained with initial pseudo-labels. 'Ours' shows our final results. (b) Efficiency and effectiveness comparison over wide range of unsupervised SOD methods the NLPR benchmark. Promoting Saliency From Depth: our approach achieves large-margin improvement over the baseline, by engaging depth information to improve pseudo-labels in the training process, without introducing additional computational cost during inference, shown red arrow.
to train the saliency prediction network ('baseline'), Moreover, the input depth map may serve as complementary source of information in refining the pseudo-labels, as it contains of spatial scene layout that may help in exposing the salient objects. Nevertheless, practical examination re- veals two main issues: (1) Inconsistency and large variations in raw depth maps: as illustrated in Fig. M (a), similar depth values are often shared by a salient object and its surrounding, making it very difficult in extracting the salient regions from depth without explicit pixel-level supervision; (2) Noises from unreliable pseudo-labels: unreliable pseudo-labels may inevitably bring false positive into training, resulting in severe damage in its prediction performance.
To address the above challenges, the following two key components are considered in our approach. First, a dispt-diientangled saliency update (DSU) framework is proposed to iteratively refine & update the pseudo-labels by engaging the depth knowledge. Here depth-disentangled network devised to explicitly learn the discriminative saliency cues and non-salient background from raw depth map, denoted saliency-guided depth sat and non-saliency- -uidded depth D Non respec- label update (DLU) module that takes advantage of
tively. This is followed by depth-disentang Dsal emphasize saliency response from pseudo-label; it also utilizes D NonSal background influence, thus facilitating more trustworthy supervision signals in training the saliency network. Note the DSU module not engaged at test time. Therefore, test time, our trained model takes input only an RGB image, instead of involving both RGB and depth as input and in the follow-up computation. Second, an attentive training strategy is introduced issue of noisy pseudo-labels; it is achieved by re-weighting the training samples in each training batch focus on those more reliable pseudo-labels. As demonstrated Fig. (b), our approach works effectively and efficiently in practice. It significantly outperforms existing unsupervised SOD methods on the widely-used NLPR benchmark. Specifically, improves over the baseline by 37%, significant amount without incurring extra computation cost. Besides, the test time execution of our approach is at 35 frame-per- per-se (FPS), the fastest among all RGB-D unsupervised methods, and par with the most efficient RGB-based methods. In summary, our main contributions as follows:
eliminate the
To our knowledge, our work is the first in exploring deep representation to tackle the prob- lem of unsupervised RGB-D saliency detection. This is enabled by two key components in the training process, namely the DSU strategy produce & refine pseudo-labels, and the attentive training strategy to alleviate the influence of noisy pseudo-labels. It results in weight architecture that engages only RGB data at test time (i.e., w/o depth map), achieving a significant improvement without extra computation cost.
Empirically, our approach outperforms state-of-the-ar the-t-r unsupervised methods on four pub- lie benchmarks. Moreover, it runs in real time 35 FPS, much faster than existing unsu- pervised RGB-D SOD methods, and at least on par with the fastest RGB counterparts.
Our approach could be adapted to work with super-sup scenario. As demonstrated in See. 4.4) augmented with our proposed DSU module, the empirical results of existing RGB-D SOD models have been notably improved.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
Remarkable progresses have been made recently in salient object detection (Ma et al. 2021 Xu et al. 2021 Pang et al. 2020b Zhao et al., 2020b Tsai et al., 2018: Liu et al. 2018). where the performance tends to deteriorate the presence of cluttered or low-contrast backgrounds. Moreover,
promising results are shown by a variety of recent efforts (Chen et al., 202 Luo et al. 2020
Chen et al.] 2020a Zhao et al. 2022, Desingh et al.] 2013; Li et al. 2020a Liao et al. 2020 Shigematsu et al.) 2017) that integrate effective depth cues to tackle these issues. Those methods, however, typically demand extensive annotations, which are labor-intensive and time-consuming.
This naturally leads to the consideration of unsupervised SODs that do not rely on such annotations.
Prior to the deep learning era (Wang et al., 2021; Zhao et al. 2020a), traditional RGB-D saliency methods are mainly based on the manually-craftes RGB features and depth cues to infer a saliency map. Due to their lack of reliance on manual human annotation, these traditional methods can be regarded as early manifestations of unsupervised SOD. Ju et al. (Ju et al., 2014) consider the in- corporation of a prior induced from anisotropic center-surround operator, and an additional depth prior. Ren et al. (Ren et al. 2013) also introduce two priors: normalized depth prior and the global- context surface orientation prior. In contrary to direct utilization of the depth contrast priors, local background enclosure prior is explicitly developed by (Feng et al. 2016). Interested readers may re- fer to (Fan et al. 2020a Peng et al. 20144 Zhou et al. (2021) for more in-depth surveys and analyses. However, by rel relying on manually craftedd priors, these methods tend to have inferior performance.
Meanwhile, considerable performance gain has been achieved by recent efforts in RGB-basee un- supervised SOD (Zhang et al. 2017 2018; Nguyen et al. 2019 Hsu et al. 20 8a). which instead construct automated feature representations using deep learning. A typical strategy is to leverage the noisy output produced by traditional methods as pseudo-label (i.e., supervisory signal) for training saliency prediction net. The pioneering work of Zhang et al. (Zhang et al.] (017) fuses the out- puts of multiple unsupervised saliency models as guiding signals in CNN training. In (Zhang al.) 2018), competitive performance is achieved by fitting a noise modeling module to the noise distri- bution of pseudo-label. Instead of directly using pseudo-labels from handerafted methods, Nguyen et al. (Nguyen et al. (019) further refine pseudo-labels via a self-supervision iterative process. Be- sides, deep unsupervised learning has been considered by (Hsu et al.. 20186 Tsai et al. 2019) for the co-saliency task, where superior performance has been obtained by engaging a fusion-learning scheme (Hsu et al.) 2018b), or utilizing dedicated loss terms (Tsai et al. 2019). These methods demonstrate that the incorporation of powerful deep neural network brings better feature represen- tation than those unsupervised SOD counterparts based on handcrafted features.
In this work, a principled research investigation on deep unsupervised RGB-D saliency detection. is presented for the first time. Different from existing deep RGB-based unsupervised SOD (Zhang et al.) 2017| 2018) that use pscudo-labels from the handcrafted methods directly, our work aims to refine and update pseudo-labels to remove the undesired noise. And compared to the refinement method (Nguyen al., 2019) using self-supervision technique, approach instead adopts disen- tangled depth information to promote pseudo-labels.
3
METHODOLOGY
3.1
OVERVIEW
Fig. 2 presents overview of our Depth-disentangled Saliency Update (DSU) framework. Overall our DSU strives to significantly improve the quality of pseudo-labels, that leads to more trustworthy supervision signals for training the saliency network. It consists of three key components. First is saliency network responsible for saliency prediction, whose initial supervision signal is provided by traditional handerafted method without using human annotations. Second, a depth network and depth-diseetangled network are designed to decompose depth cues into saliency-guided depth D and con-saliency-guidde depth DNonSal, explicitly depict saliency cues in the spatial layout. Third, a sogph--disnttngledd label update (DLU) module is devised to refine and update the pseudo- labels, by engaging the learned Di and DNonSul- The updated pseudo-labels could in turn provide more trustworthy supervisions for the saliency network. Moreover, an attentive training strategy (ATS) is incorporated when training the saliency network, tailored for noisy unsupervised learning by mitigating the ambiguities caused by noisy pseudo-labele We note that the DSU and ATS are not performed at test time, so does not affect the inference speed. In other words, the inference stage
In this paper, unsupervised learning refers to learning without using human annotation (Zhag al. 2017
3

---

Published as a conference paper at ICLR 2022
Salleney Networt:
Sul
Element-wise Addition
pAGa
Element-wiae Subtraction
Flement-wise Multiplicarion
le
Updated
Holistic Atlention
Pseado Label
Updatei
Upline3
Reffermene
Supervision Signal
Labed
Depth-dieentangled
Update
referted Section 3.2.1.
Deyth Map
RGB
Coran 1:
CSN
Dsot
Consistency Lews
Dopth Network
Depth Map
R.
from
lga
Daua
H
Eq.
pDrpen
Dh D
fobepek
Solpaat
CSN
Desaial
Danatal
Depth disentangled Network
Figure 2: Overview of the proposed method. The saliency network is trained with the iteratively updated pseudo-labels. The depth network and depth-disentangled network are designed decom- pose raw depth into sefiency-guidee depth Dsal and non www-ssclllleuuuiaeee are subsequently fed into the sopth-disentangled label update (DLU) module refine and update pseudo-labels. The inference stage involves only the black dashed portion.
depth Non which
involves only the black dashed portion the proposed network architecture in Fig. 2 i.e., only RGB images are used for predicting saliency, which enables very efficient detection.
3.2 DEPTII-DISENTANGLED SALIENCY UPDATE
The scene geometry embedded in the depth map can serve complementary information source to refine the pseudo-labels. Nevertheless, as diseussed the introduction section, direct adoption of the raw depth may not necessarily lead to good performance due to the large value variations and inherit noises in raw depth map. Without explicit pixel-wise supervision, the saliency model may unfortunately be confused by nearby background regions having the ent contexts; meanwhile, depth variations salient or background regions may also result in false responses. These observations motivate to design capture discriminative saliency cues from depth, and a depth--isentangled label update (DLU) strat- egy to improve and refine pseudo-labels by engaging the disentangled depth knowledges.
depth but with differ-
dipph--dsentnngled network effectively
3.2. DEPTII-DISENTANGLE NETWORK
The depth-disentangled network aims capturing valuable saliency as well redundant non-salient cues from raw depth map. As presented in the bottom of Fig. 2 informative depth feature Depth first extracted from the depth network under the supervision of raw depth map, using the mean square error (MSE) loss function, i.., la,1- The DDepth is then decomposed into saliency-guided depth D.at and non-saliency-guiee ing the model to learn saliency-specio cues from depth; 2) ensuring the coherence between the disentangled and original depth features.
depth DNouSal following two principles: 1) explicitly guid-
Specifically, in the bottom right of Fig. 2 we first construct the spatial supervision signals for the network. Given the rough saliency prediction Salpred from the saliency network and the raw depth map Dmap- the (non-)saliency- can be obtained by multiplying Salpred (or Salprea) and depth map D Dmap in spatial attention manner. Since the predicted saliency may contain errors introduced from the inaccurate pseudo- labels, employ holistic attention (HA) operation (Wu et al. 2019) smooth the coverage of the predicted saliency, so as to effectively perceive more saliency area from depth. Formally, the (non-) Jsaassssyagguiiiieeee depth masks generated by:
depth masks, i.e., DL.aeel and DLabel NonSal Sal
D Sal
V max (Sal k), Salpred) Dmaps
D Labet V Vmax (Fa - Salpred; k), Sall @ Dmap:
(2)
where Fo(-k) represents the HA operation, which is implemented using the convolution operation with Gaussian kernel and zero bias; the size and standard deviation of the Gaussian kernel are initialized with 32 and 4, respectively, which are then finetuned through the training procedure;
4

---

Published as conference paper at ICLR 2022
Image
Depth
Initial Label
D
Duot
Divedat
Updated Label
GT
Figure 3: The internal inspections of the proposed DSU. is observed that the updated label exhibits more reliable saliency signals than initial pseudo-label.
V maxx( (.).) ) is a maximum function to preserve the higher values from the Gaussian filtered map and the original map; denotes pixel-wise multiplication.
NonSut- D is fed into D-Sal CNN and D-NonSal CNN explicitly learn valuable saliency and redundant non-salient cues from depth map, generat- ing Dsat and D Nonsall respectively. The loss functions here (i.e. [sat and NNmsaal) ) are MSE loss. Detailed structures of D-Sal and D-NonSal CNNs are in the appendix. To further ensure the co- herence between the disentangled and original depth features, a consistency loss, Lcons is employed as:
Building upon the guidance of DLabel and D Labee
W C ΣΣΣ
H
Icon HXW X
Depth Lijk
F.Depth J.j. 112
(3)
3= R=1
where F:Depth is the sum the disentangled F FDDepth and E>Depth which denotes the regenerated depth feature; H, W and C are the height, width and channel of PDepth and i:Dephh |I-ll represents Euclidean norm. Here, Depth is also under the supervision of depth map, using MSE loss, i.e., la22-
Then, the overall training objective for the depth network and the depth-disentangledd network is as:
1
deepth
(12,1 + Saa NonSal + Xl
5N
where denotes the nech sample in mini-batch with N training samples; À is set 0.02 in the experiments to balance the consistency loss lcon and other loss terms. As empirical evidences in Fig. B suggested, comparing the raw depth map, Ds better at capturing discriminative cues of the salient objects, while NonSall better depicts the complementary background context.
In the next step, the learned Dsat and DoonSal are fed into our Depth-disentangled Label Update, to obtain the improved pseudo-labels.
3.2.2 DEPTH-DEE DISENTANGLED LABEL UPDATE
To maintain more reliable supervision signals in training, a depth-disentangled label update (DLU) strategy devised to iteratively refine & update pseudo-labels. Specifically, as shown in the upper stream of Fig. 2 using the obtained Sal D and D) NonSSal- the DLU simultaneously highlights the salient regions the coarse saliency prediction by the sum Salpred and D and suppresses the non-saliency negative responses by subtracting NonSal in pixel-wise manner. This process can be formulated as:
Stemp
Saftá Por + D'u
D NanSal
(5)
To avoid the value overflow of the obtained Stemp (i.e., removing negative numbers and normalizing the results to the range of [0, 1]), a thresholding operation and a normalization process are performed as:
S. min)
fo. si. Stmpp
sij
< .i E W], (6) temp
SN mar(S.)) min where S,
others
where min(.) and (axx()) denote the minimum and maximum functions. Finally, fully- fully-connected conditional random field (CRF (Hou et al., (007)) is applied to Sx, to generate the enhanced saliency map Smap the updated pseudo-labels. The empirical result of the proposed DSU step is
showcased in Fig. 3 with quantitative results presented in Table These internal evidences suggest that the DSU leads noticeable improvement after merely several iterations.
5

---

Published as a conference paper at ICLR 2022
3.3 ATTENTIVE TRAINING STRATEGY
When training the saliency network using pseudo-labels, an attentive training strategy (ATS) is pro- posed tailor for the deep unsupervised learning context, reduce the influence of ambiguous pseudo-labels, and concentrate on the more reliable training examples. This strategy is inspired by the human learning process of understanding new knowledge, that is, from general to specific under- standing cycle (Dixon 1999 Peltier et al., 2005). The ATS alternaees between two steps to re- re-weigh the training instances in a mini-batch.
To be specific, we first start by settling the related loss functions. For the nea sample in a mini-bated with N training samples, we define the binary cross-entropy loss between the predicted saliency Salfred and the pseudo-label S
map
I, -(Shaa log Sal pred + - (1-Smap) log(1 Salgred))
Then, the training objective for the saliency network in current mini-batch is defined as an attentive binary cross-entropy loss Leal, which can be represented as follows:
Leal
1
N C(a
(an-
In), an
1. Eien e4 Eier efit
step one,
Σ
ON
n=1
step two,
where a, represents the weight of the nus training sample at current training miti-batch.
The ATS starts from a uniform weight for each training sample in step one to learn general representations across a lot of training data.
0.24
Bactttone unfoom weight Boctttone attentive weight
0.23
0.22
(Backbone-
Step two decreases the importance of ambigu- ous training instances through the imposed at- tentive loss; the higher the loss value, the less weight an instance is to get.
0.21
0.
0.99
0.58
0.17
In this paper, we define step one and two training round (27 epochs). During each train- ing round, the saliency loss Leal and depth loss Ldepth are optimized simultaneously to train their network parameters. The proposed DLU is taken at the end of each training round to up- date the pseudo-labels for the saliency network: meanwhile, D Sat and D Diaaeet NonSal in E and 2 are also updated using the improved palpred- This allows the network to make efficient use of the updated pseudo labels. As suggested by Fig. and Ta the use of our attentive training strategy leads to more significant error reduction compared with the backbone solely using uniform weight or attentive weight for each instance. Besides, when our DSU and the attentive training strategy are both incorporated to the backbone, better results are achieved.
0.16
0.M4
0.13
8 15
24 Epoch
Figure 4: Analysis of the proposed attentive train- ing strategy. when evaluated NJUD testset. The 'backbone" refers to the saliency network trained with initial pseudo-labels.
4 EXPERIMENTS AND ANALYSES
4.1 DATASETS AND EVALUATION METRICS
Extensive experiments are conducted over four large-scale RGB-D SOD benchmarks. NJUD (Ju et al. 2014) in its latest version consists of 1,985 samples, that are collected from the Internet and 3D movies; NLPR (Peng et al. 2014) has 1,000 stereo images collected with Microsoft Kinect; STERE Niu al. 2 contains 1,000 pairs of binocular images downloaded from the Internet; DUTLF-Depth (Piao et al. 2019) has 1,200 real scene images captured by a Lytro2 camera. We follow the setup of (Fan et al. 2020a) construct the training set, which includes 1,485 samples from NJUD and 700 samples from NLPR, respectively. Data augmentation is also performed by randomly rotating, cropping and flipping the training images to avoid potential overfitting. The
remaining images are reserved for testing.
Here, five widely-used evaluation metrics are adopted: E-measure (Eş) FFnn al. 2018 weighed F-measure (FF") (Margolin et al., 2014), F-measure (Fa) (Achanta et al. 2009). ). Mean Absolute Error (MAE or M) (Borji et al. 20155). and inference time(s) or FPS (Frames Per Second).
Implementation details are presented in the Appx A.3 Source code is publicty available,

---

Published as a conference paper at ICLR 2022
Image
Depth
GT
Ours DeepUSPS CDCP
DCMC
ACSD)
DSR
BSCA
RBD
Figure 5: Qualitative comparison with unsupervised saliency detection methods. GT denotes ground-truth.
Table I: Quantitative comparison with unsupervised SOD methods. 'Backbone' refers to the saliency feature extraction network (Wu et al. 2019) adopted in our pipeline, i.e. the one without the two proposed key components. The RGB- based methods are specifically marked by T. UnSOD is shorthand for unsupervised
SOD, We also provide the results of existing fully supervised methods that can be referenced in Table
Inference Timets))
NJUD
NLPR
STERE
DUTL.-Depth F3
Eg]
Fg
Fa
MM
Eg
P3
F31
Eg
F7 Fg
Fa
RBD
Zha
2014
0.189
.684
.387
.556
.256
.765
.388
.500
.211
.730
.443
.610
225
.733
.447
.619
.222
Handoraffed UnSDD
MST
2015
0.030
.670
.291
.436
288
.762
.257
A91
.99
.681
.312
.447
269
6788
.254
.401
.279
BSCA
26665
.7%6
.554
DSR
Lie
2013
0.376
.759
195
7757
.130
785
.486
7577
.164
ACSD
0.118
790
195
751
327
.547
.171
793
.225
66
250
2.10
185
868
DES
2014
7.790
.421
.241
.165
.488
725
229
-533
.301
.675
.383
.992
29/
7733
.356
.655
.280
I.H
Ponr ot 10
2014
2130
772
.111
625
201
772
3330
530
119
772
360
785
171
7767
.350
.699
1774
GP
RC
12.98
7700
.325
866
204
813
347
.70
144
785
371
710
182
CDB
201
0.600
752
.A18
655
200
.110
356
.628
108
808
436
713
166
SF:
10 2016
1.570
780
518
735
164
853
578
701
1885
825
546
747
145
730
330
.474
.95
DCMC
2010
1,210
795
-555
684
265
322
.966
.82
.529
7743
148
.712
.406
.243
MB
201773
CDCF
Phirera 20176
5.720
754
.150
Deep UnSDD
USD! 7hane et al. 2018
0.0180
7668
565
6630
163
776
.336
.50
.119
796
.572
670
146
.705
.545
.650
.157
DocpUSPS [Ngiyar anr 2019
0.0292
.058
.140
Rockhene gains
0.0286
759
5500
794
555
166
16665
Ours
0.0266
.797
597
.719
745
.665
.678
.774
.854
.650
.107
4.2 COMPARISON WITH THE THE
Our approach is compared with 15 unsupervised SOD methods, i.e,, without using any human anno- tations. Their results are either directly furnished by the authors of the respective papers, or gener- ated by re-running their original implementations. In this paper, we make the first attempt to address deop-learning-basee -bsed unsupervised RGB-D SOD. Since existing unsupervised RGB-D methods are all based on handcrafted feature representations, we additionally provide several RGB-based meth- ods (e.g., USD and DeepUSPS) for reference purpose only. This gives more observational evidences for the related works. These RGB-based methods are specifically marked by f in Table
Quantitative results are listed in Table where our approach clearly outperforms the state-of-the- art unsupervised SOD methods in both RGB-D and RGB only scenarios. This due to our DSU framework that leads to trustworthy supervision signals for saliency network. Furthermore, our network design leads to a light-weight architecture in the inference stage, shown as the black dashed portion in Fig. 2 This enables efficient & effective detection of salient objects and brings large- margin improvement over the backbone network without introducing additional depth input and computational costs, as shown in Fig. I and Ta Qualitatively, saliency predictions of competing methods are exhibited in Fig. 5 These results consistently proves the superiority of our method.
4.3
ANALYSIS OF THE EMPIRICAL RESULTS
The focus here is on the evaluation of the contributions from each of the components, and the evaluation of the obtained pseudo-labels as intermediate results.
Effect of each component. In Table 2 we conduct ablation study to investigate the contribution of each component. To start with, we consider the backbone (a), where the saliency network trained with initial pseudo-labels. As our proposed ATS and DSU are gradually incorporated, increased performance has been observed on both datasets. Here we first investigate the benefits of ATS by applying it to the backbone and obtaining (b). We observe increased F-meauure scores of 3% and 5.8% on NJUD and NLPR benchmarks, respectively. This clearly shows that the proposed ATS can effectively improve the utilization of reliable pseudo-labels, by re- se-weighting ambiguous training data. We then investigate in detail all components in our DSU strategy. The addition of the entire DSU leads (f), which significantly improves the F-measure metrie by 11. 11.3% and 23.5% 5% on each
7

---

Published as a conference paper at ICLR 2022
of the benchmarks, while reducing the MAE by 22.4% and 41.9%, respectively. This verifies the effectiveness of the DSU strategy to refine and update pseudo-labels. Moreover, as we gradually exclude the consistency loss Loon (row (c)) and HA operation (row (d)), degraded performances are observed on both datasets. For an extreme case where we remove the DLU and only maintain CRF to refine pseudo-labelss, it is observed that much worse performance is achieved. These results consistently demonstrate that all components in the DSU strategy are beneficial for generating more accurate pseudo-labels.
We also display the visual ev- idence of the updated pseudo- labels obtained from the DSU
Table 2: Ablation study our deep unsupervised RGB-D SOD pipeline,
using the F-measure and MAE metrics.
NJUD
NIPR
strategy in Fig. 6 is shown
Index
Midel Setups
Fs
M
Fg
M
that the initial pseudo-labe labels un-
(a)
Backbone (a) attentive training strategy
0.627
0.186
0.570
0.126
(6)
0,646
0.174
0.603
0.112
fortunately tend to miss impor-
(C)
(b) + CRF
0.674
0.150
0.663
0.093
tant parts as well as fine-grained
(d)
(b) DSC (w/o (
0.703
0.141
0.716
0.074
details. The application of CRF
(c)
DSU strategy
(b) + DSU (ea)
0.712
0.137
0.735
0.068
(b) + DSU (Ours)
0.719
0.135
0.745
0.065
helps to filter away background noises, while salient parts could still be missing out. By adopting our DSU and attentive training strategy, the missing parts could be retrieved in the updated pseudo-labels, with the object silhouette also being refined. These numerical and visual results consistently verify the effectiveness of our pipeline in deep unsupervised RGB-D saliency detection.
Detailed analysis of the ATS. The influence of the attentive training strategy (ATS) is stud- ied in detail here. The error reduction curves in Fig. 4 have shown that the use of ATS could lead to greater test error reduc- tion compared with the backbone that uses uniform weight or at- tentive weight for each instance. Table the ATS also enables the network to learn more efficiently than using uniform weight or attentive weight alone, by comparing 'Ours' with 'Setting 1' and 'Setting 2'. In addition, we discuss the effect of different alternation intervals 7 in ATS. As listed in Table the larger smaller interval leads to inferior performance due to the insufficient or excessive learning of saliency models.
Table 3: Analyzing attentive training strategy (ATS) with different set- tings. 'Setting 1' backbone + uniform weight + DSU, and 'Setting 2' is backbone + attentive weight + DSU. The last four columns show backbone + ATS + DSU with different alternation interval r.
Analysis of the ATS
Ours
Analysis of the interval r
Sctting
Setting 2
7 ==3
==1
1
==5
t r=l 10
Fg
0.695
0.707
0.719
0,711
0,712
0.687
NJUD
M
0.148
0.142
0.135
0.143
0118
0.147
F5
0.706
0.737
0.745
0.742
0.739
0.698
NLPR
M1
0.071
0.067
0.065
0.069
0.066
0.073
When combining the ATS with the DSU strategy as shown in
Table 4: Internal mean absolute errors, each is evaluated between current pseudo-labels and the corresponding true labels (only used for evaluation purpose) during the train- ing process.
Pscudo label Update
Initial
Update 1
Update 2
Update 3
Update
Moun absolute erooe
0.162
0.124
0.117
0.116
0.116
Table 5: Comparison of different pseudo-label generation variants. 'CRF' refers to fully-connected CRF. "OTSU" represents the standard Otsu image thresholding method.
Imuge
Initial
-CRF
Update
Updint
Figure 6: Visual examples of the intermedi- ate pseudo-labels used in our approach. 'Initial' shows the initial pseudo-labels generated by tra- ditional handcrafted method, '+CRF' refers to the pscudo-labels after applying fully fulle-con CRF. Update 1&2 represent the updated pseudo- labels produced in our pipeline over training rounds. "GT" means the ground truth, used for reference purpose only.
Label Accuray
Eş 1
Fy f
Fa f M
Initial pseudo-label
0.760
0.526
0,614
0.162
Initial pseudo-label CRF
0.763
0.578
0.634
0.144
Our DSU
0.792
0.635
0.708
0.116
Depth map Depth map OTSU
0.419
0.284
0.164
0.414
0.465
0.398
0.429
0.332
Analysis of labuds-labels. We analyze the quality of pseudo-labels over the training process in Ta- 4 where the mean absolute error scores between the pseudo-labels at different update rounds and the ground-trutt -ouund-truth labels are reported. is observed that the quality of pseudo-labels is significantly improved during the first two rounds, which then remains stable in the consecutive rounds. Fig. 6 also shows that the initial pseudo- is effectively refined, where the updated pseudo-label is close to the true label. This provides more reliable guiding signals for training the saliency network.
8

---

Published as a conference paper at ICLR 2022
Comparison of different pseudo-label variants. In Table 5 we investigate other possible pseudo- label generation variants, including, initial paeudo-label with CRF refinement, raw depth map, and raw depth map together with OTSU thresholding (Otsu 1979.. It is shown that, compared with the direct use of CRF, our proposed DSU is able to provide more reliable pseudo-labels, by disentangling depth promote saliency. It is worth noting that a direct application of the raw depth map or together with an OTSU adaptive thresholding of the depth map, may nevertheless lead to awful results. We conjecture this is because of the large variations embedded in raw depth, and the fact that foreground objects may be affected by nearby background stuffs that are close in depth.
Is repth-disentangleme
necessary? In our DSU, we disentangle the raw depth map into valuable salient parts as
Table 6: Discussion on different pseudo-label updating settings.
NJUD
NLPR
well as redundant non-salient areas to simultaneously highlight the salient re- gions and suppress the non-salient neg- ative responses. To verify the necessity of depth disentanglement, we remove the NonSaal branch in the depth disentangled network and only use the Dsai to update pseudo-labels in DLU. Empirical results in rable| that removing D NonSaa (row (a)) leads to significant inferior performance when comparing with the original DSU (row (c)), which proves the superiority of our DSU design.
Different leeudo-label Updating Setups
Fg 1
M
Fat
M
DSU w/o D NanSool
0.698
0.143
0.715
0.072
(b) DSU using D NanSat
0.691
0.154
0.707
0.079
(0) DSU using Diat NomSall (Ours)
0.719
0.135
0.745
0.065
How about using raw depth labels in DSU? We also consider directly utilize the raw disen- tangled depth labels, i.e., Deabei and Daaie to update pseudo-labels in DSU. However, direct adoption of the raw depth labels does not lead to good performance, as empirically exhibited in (b). The DSU using raw labels performs worse compared to our original DSU design in row (c). This is partly due to the large value variations and intrinsic label noises in raw depth map as discussed before. On the other hand, benefiting from the capability of deep-learning brarminng-asedd networks to learn general representations from large-scale data collections, using Dsat and NonsSal in our DSU strategy is able to eliminate the potential biases in single raw depth label. Visual evidences as cxhibited in the 4th-7th columns of Fig. 3 also support our claim.
4.4 APPLICATION TO FULLY-SUPERVISED SETTING
To show the generic applicability of our approach, a variant of our DSU is applied to several cutting- edge fully-supervised SOD models to improve their performance. This is made possible by re- defining the quantities Dabbll = Scr & Dmap and D (1 Scr) 8 Dmap. with Ser being the ground truth saliency. Then the saliency net- work (i.e., existing SOD models) and the depth- disentangled network are retrained by Sor and the new Diabel and D Deabet Nonsaa respectively. After training, the proposed DLU is engaged to obtain the final improved saliency. In Table 7 we report the original results of four SOD methods and the new results of incorporating our DSU strategy on two popular benchmarks. is observed that our supervised variants have consistent performance im- provement comparing to each of existing models. For example, the average MAE score of four SOD methods on NJÜD benchmark is reduced by 18.0%. We attribute the performance improvement to our DSU strategy that can exploit the learned Dzat to facilitate the localization of salient object regions in a scene, as well as suppress the redundant background noises by subtracting DonsSll- More experiments are presented in the Appx A.2
Table 7: Applying our DSU to existing fully- supervised RGB-D D SOD methods.
NJUD
NLPR
Pg f
M 1
Fg 1
M
DMRA (Piao et al.
2019
0.872
0.051
0.855
0.031
+ur
0.893
0.044
0.879
0.026
CMWN (Li at 2020c
0.878
0.047
0.859
0,029
Our DSU
0.901
0.041
0.882
0.025
FRDT (Zhan al.
20201
0.879
0.048
0.868
0.029
+Oir DSU
0.903
0.038
0.901
0.023
CPD Wuu al.
2019
0.873
0.045
0.866
0.028
+Orr DSU
0.909
0.036
0.907
0,022
CONCLUSION AND OUTLOOK
This paper tackles the new task of deep unsupervised RGB-D saliency detection. Our key insight is to internally engage and refine the pseudo-labels. This is realized by two key modules, the depth- disentangled saliency update in iteratively fine-tuning the pseudo-labels, and the attentive training strategy in addressing the issue of noisy -seudo-labels. Extensive empirical experiments demon- strate the superior performance and realtime efficiency of our approach. For future work, we plan to extend our approach to scenarios involving partial labels.
Acknowledgement. This research was partly supported by the University of Alberta Start-up Grant, UAHJIC: Grants, and NSERC Discovery Grants (No. RGPIN-2019-04575)

---

Published as a conference paper at ICLR 2022
REFERENCES
Radhakrishna Achanta. Sheila Hemami, Francisco Estrada, and Sabine Susstrunk. Frequency- salient region detection. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 1597-1604, 2009.
Qi Bi, Shuang Yu, Wei Ji, Cheng Bian, Lijun Gong, Hanruo Liu, Kai Ma, and Yefeng Zheng. Local-global dual perception based deep multiple instance learning for retinal disease classification. In International Conference on Medical Image Computing and Computer-Assister Intervention, pp. 55-64. Springer, 2021.
Qi Bi, Beichen Zhou, Wei Ji, Kun Qin, Qinghao Ye, and Gui-Song Xia. All grains, one scheme (agos); Learning multi-grain instance representation for acrial scene classification. IEEE Transactions on Image Processing. 2022.
Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: A benchmark. IEEE Transac- tions on Image Processing, 24(12):5706 -7222 2015.
John Canny. computational approach to edge detection, IEEF Transactions on Pattern Analysis and Machine Intelligence, 8(6):79-698, 1986.
Chenglizhao Chen, Jipeng Wei, Chong Peng, Weizhong Zhang, and Hong Qin. Improved saliency detection in RGB-D images using two-phase depth estimation and selective deep fusion. IEEE Transactions on Image Processing, 29:4296-4300 2020a.
Chenglizhao Chen, Jipeng Wei, Chong Peng, and Hong Qin. Depth-quality-aware salient object detection. IEEE Transactions on Image Processing, 30:2350-2363, 2021a.
Hao Chen and Youfu Li. Progressively complementarity-oo aware fusion network for RGB-D D salient object detec- tion. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3051-3060, 2018.
Hao Chen and Youfu Li. Three-stream attention-aware network RGB-D salient object detection. JEEE Transactions on Image Processing, 2866)22255 2019.
Hao Chen, Youfu Li, and Dan Su. Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for RGB-D salient object detection. Pattern Recognition, 86:376-385, 2019.
Qian Chen, Ze Liu, Yi Zhang, Keren Fu, Qijun Zhao, and Hongwei Du. RGB-D salient object detection via 3D Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021b.
convolutional neural.
Shuhan Chen and Yun Fu. Progressively guided alternate refinement network for RGB-D salient object detec- tion. In Proceedings of the European Conference on Computer Vision, 2020.
Wenting Chen, Shuang Yu, Kai Ma. Wei Ji, Cheng Bian, Chunyan Chu, Linlin Shen, and Yefeng Zheng. Tw-gan: Topology and width aware gan for retinal artery/vein classification. Medical Image Analysis, pp. 102340, 2021c.
Zuyao Chen, Runmin Cong, Qianqian Xu, and Qingming Huang. DPANet:
Depth potentiality-aware gated attention network for RGB-D salient object detection. IEEE Transactions on Image Processing, 2020b.
Yupeng Cheng, Huazhu Fu, Xingxing Wei, Jiangjian Xiao, and Xiaochun Cao. Depth enhanced saliency detec- tion method. In Proceedings of International Conference Internet Multimedia Computing and Service, pp. 23 23-27, 2014.
Runmin Cong, Jianjun Lei, Changqing Zhang, Qingming Huang, Xiaochun Cao, and Chunping Hou, Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion. IEEE Signal Processing Letters, 23(6):819-221 2016.
Karthik Desingh, K Madhava Krishna, Deepu Rajan, and CV Jawahar. Depth really matters: Improving visual salient region detection with depth. In British Machine Vision Conference, 2013.
Nancy M. Dixon. The organizational learning cycle: How we can learn collectively. Gower Publishing, Ltd., 1999.
Deng-Ping Fan. Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, and Ali Borji. Enhanced-alignment measure for binary foreground map evaluation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 698-704, 2018.
10

---

Published as a conference paper at ICLR 2022
Deng-Ping Fan, Zheng Lin, Zhao Zhang, Menglong Zhu, and Ming-Ming Cheng. Rethinking RGB-D salient object detection: Models, data sets, and large-scale benchmarks. IEEE Transactions on Neural Networks and Learning Systems, 2020a.
Deng-Ping Fan, Yingjie Zhai, Ali Borji, Jufeng Yang. and Ling Shao. BBS-Net: RGB-D salient object detection with bifurcated backbone strategy network. Proceedings of the European Conference on Computer Vision, 2020b.
David Feng, Nick Barnes, Shaodi You, and Chris McCarthy. Local background enclosure for RGB-D salient object detection. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2343-2350, 2016.
Keren Fu, Deng-Ping Fan, Ge-Peng Ji, and Qijun Zhao. JL-DCF: Joint learning and densely-cooperraive fusion framework for RGB-D salient object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3052- 3062, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning, pp- 1050-1059, 2016.
Silvio Giancola, Matteo Valenti, and Remo Sala. A Survey 3D Cameras: Metrological Comparison of Time-of-Fligge Flight, Structured- Light and Active Stereoscopy Téchnologies. Springer, 2018
Jingfan Guo, Tongwei Ren. and Jia Bei. Salient object detection for RGB-D image via saliency evolution. In IEEE International Conference on Multimedia and Expo, pp. 1-6, 2016.
Junwei Han, Hao Chen, Nian Liu, Chenggang Yan, and Xuelong Li. CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion. IEEE Transactions on Cybernetics, 48( (111)1111111111 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedingy of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip HS Torr. Deeply supervised salient object detection with short connections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3203-3212, 2017.
Kuang-Jui Hsu, Yen- Lin, Yung-Yu Yu Chuang, et al. Co-attention enns for unsupervised object co- segmentation. In IJCAI, pp. 748-756, 2018a.
Kuang-Jui Hsu, Chung-Chi Tsai, Yen-Yu Lin, Xiaoning Qian, and Yung- Yu Chuang. Unsupervised enn-based co-saliency detection with graphical optimization. In European Conference on Computer Vision, pp. 485- 501, 2018b.
Wei Ji, Wenting Chen, Shuang Yu, Kai Ma, Li Cheng, Linlin Shen, and Yefeng Zheng. Uncertainty quan- tification for medical image segmentation using dynamic label factor allocation among multiple raters. In MICCAI on QUBIQ Workshop, 2020a.
Wei Ji, Jingjing Li, Miao Zhang, Yongri Piao, and Huchuan La. Accurate RGB-D salient object detection via collaborative learning. In Proceedings of the European Conference on Computer Vision, pp. 52-69, 2020b.
Wei Ji, Jingjing Li. Shuang Yu, Miao Zhang, Yongri Piao. Shunyu Yao, Qi Bi, Kai Ma, Yefeng Zheng, Huchuan Lu, and Li Cheng- Calibrated rgb-d salient object detection. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 9471-9481, 2021a.
Wei Ji. Shuang Yu, Junde Wu, Kai Ma. Cheng Bian, Qi Bi, Jingjing Li, Hanruo Liu, Li Cheng, and Yefeng Zheng. Learning calibrated medical image segmentation via multi-rater agreement modeling. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 12341-12351, 2021b.
Ran Ju, Ling Ge, Wenjing Geng, Tongwei Ren, and Gangshan Wu. Depth saliency based on anisotropic center- surround difference. IEEE International Conference on Image Processing, pp. 1115 1119, 2014.
ByoungChul Ko, Soo Yeong Kwak, and Hyeran Byun. SVM-based salient region(s) extraction method for image retrieval. In Proceedings of the 17th International Conference Pattern Recognition, volume 2, pp. 977-980, 2004.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Chongyi Li, Runmin Cong, Sam Kwong, Junhui Hou, Huazhu Fu, Guopu Zhu, Dingwen Zhang, and Qingming Huang. ASTF-Net: Attention steered interweave fusion network for RGB-D salient object detection. IEEE Transactions on Cybernetics, 1113 2020a.
11

---

Published as a conference paper at ICLR 2022
Chongyi Li, Rummin Cong, Yongri Piao, Qianqian Xu, and Chen Change Loy. salennt object detection with modality modulation and selection. In Proceedings of the European Conference on Computer Vision, 2020b.
Gongyang Li, Zhi Liu, Linwei Ye, Yang Wang, and Haibin Ling. mross-modal weighting network for RGB-D salient object detection. In Proceedings of the European Conference on Computer Vision, 2020c.
Gongyang Li. Zhi Liu, Minyu Chen, Zhen Bai, Weisi Lin, and Haibin Ling. Hierarchical alternate interaction network for RGB-D D salient object detection. IEEE Transactions Image Processing, 2021a.
Jingjing Li, Wei Ji, Qi Bi, Cheng Yan, Miao Zhang, Yongri Piao, Huchuan Lu, et al. Joint semantic mining for weakly supervised rgb-d salient object detection. Advances in Neural Information Processing Systems, 34, 2021b.
Xiaohui Li, Huchuan Lu, Lihe Zhang, Xiang Ruan, and Ming-Hssun Yang. Saliency detection via dense and sparse reconstruction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2976- 2983, 2013.
Fangfang Liang, Lijuan Duan, Wei Ma, Yuanhua Qiao, Zhi Cai, and Laiyun Qing. Stereoscopic saliency model using contrast and depth-guides guuded-backgroume prior. Neurocomputing, 275:2227-225 222288 2018.
Guibiao Liao, Wei Gao. Qiuping Jiang, Ronggang Wang, and Ge MMNet: Multi-stage and multi-scale sion network for RGB-D salient object detection. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 2436-2444, 2020.
Huiying Liu. Shugiang Jiang, Qingming Huang, and Changsheng Xu. A generic virtual content insertion system based on visual attention analysis. In Proceedings of the 16th ACM International Conference on Multimedia, pp. 379-388, 2008.
Jiang-Jiam Liu, Qibin Hou, Ming-Ming Miig Cheng, Jiashi Feng, and Jianmin Jiang. A simple pooling-bacc based design for real-time salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3917-3926, 2019.
Nian Liu, Junwei Han, and Ming- Hsuan Yang. Picanet: Learning pixel-wise contextual attention for saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3089 3098, 2018
Nian Liu, Ni Zhang, and Junwei Han. Learning selective self-mutual attention for RGB-D saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 13756-13766 13765, 2020.
Ao Luo, Xin Li, Fan Yang, Zhicheng Jiao, Hong Cheng, and Siwei Lyu. Cascade graph neural networks for RGB-D salient object detection. In Proceedings of the European Conference on Computer Vision, 2020.
Mingean Ma, Changqun Xia, and Jia Li. Pyramidal feature shrinking for salient object detection. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pp. 2311-23111 1-2118, 2021.
Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal. How to evaluate foreground maps? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2014.
Tam Nguyen. Maximilian Dax, Chaithanya Kumar Mummadi, Nhung Ngo, Thi Hoai Phuong Nguyen, Zhongyu Lou, and Thomas Brox. DeepUSPS: Deep robust unsupervised saliency prediction via self- supervision. In Advances in Neural Information Processing Systems, pp. 204-214, 2019.
Munan Ning, Cheng Bian, Donghuan Lu, Hong-Yu Yu Zhou, Shuang Yu, Chenglang Yuan, Yang Guo, Yaohua Wang, Kai Ma, and Yefeng Zheng. A macro-micro weakly framework for as-oct tissue segmen- tation. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pp. 725 734. Springer, 2020.
Munan Ning, Cheng Bian, Dong Wei, Shuang Yu, Chenglang Yuan, Yaohua Wang, Yang Guo, Kai Ma, and Yefeng Zheng. A new bidirectional unsupervised domain adaptation segmentation framework. In Interna- tional Conference on Information Processing in Medical Imaging, pp. 492-503. Springer, 2021a.
Munan Ning, Cheng Bian, Chenglang Yuan, Kai Ma, and Yefeng Zheng. Ensembled resunet for anatomi- cal brain barriers segmentation. Segmentation, Classification, and Registration of Multi-modality Medical Imaging Data, 12587:27, 2021b.
Munan Ning. Donghuan Lu, Dong Wei, Cheng Bian, Chenglang Yuan, Shuang Yu, Kai Ma. and Yefeng Zheng Multi-anchor active domain adaptation for semantic segmentation. In Proceedings of the IEFE International Conference on Computer Vision, 2021c.
12

---

Published as a conference paper at ICLR 2022
Yuzhen Niu, Yujie Geng, Xueqing Li, and Feng Liu, Leveraging stereopsis for saliency analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 454 461, 2012.
Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1):22-66 66, 1979.
Youwci Pang, Lihe Zhang, Xiaoqi Zhao, and Huchuan Lu. Hicrarchical dynamic filtering network for RGB-D salient object detection. In Proceedings of the European Conference on Computer Vision 2020a.
Youwei Pang, Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu. Multi-scale interactive network for salient object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9413- 9422, 2020b.
James Peltier, Amanda Hay, and William Drago. The reflective learning continuum: Reflecting on reflection. Journal of marketing education, 27(3):50--633, 2005.
Houwen Peng, Bing Li, Weihua Xiong, Weiming Hu, and Rongrong Ji. RGBD salient object detection; benchmark and algorithms. In Proceedings of the European Conference Computer Vision, pp. 92 109, 2014.
Yongri Piao, Wei Ji, Jingjing Li, Miao Zhang, and Huchuan Depth- multi scale recurrent attention network for saliency detection. In Proceedings of the IEEE International Conference on Computer Vision pp. 7254-7263, 2019.
Yongri Piao, Zhengkun Rong, Miao Zhang, Weisong Ren, and Huchuan Lu. A2dele: Adaptive and attentive depth distiller for efficient RGB-D salient object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 9060 9069, 2020.
Yao Qin, Huchuan Lu, Yiqun Xu, and Wang. Saliency detection via cellular automata. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 110-119, 2015.
Lianggiong Qu, Shengfeng He, Jiawei Zhang, Jiandong Tian, Yandong Tang, and Qingxiong Yang. RGBD salient object detection via deep fusion. IEEE Transactions Image Processing, 26(5):2244-2883, 2017.
Jianqiang Ren, Xiaojin Gong, Lu Yu, Wenhui Zhou, and Michael Ying Yang. Exploiting global priors for RGB- D saliency detection. Proceedings of the IEFF Conference Computer Vision and Pattern Recognition Workshops, pp. 25-32, 2015.
Ling Shao and Michael Brady, Specific object retricval based on salient regions. Pattern Recognition, 39(10): 1932 1948, 2006.
Riku Shigematsu, David Feng. Shaodi You, and Nick Barnes. Learning RGB-D salient object detection using background enclosure, depth contrast, and top-down features. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 2749-2757, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Peng Sun, Wenhu Zhang, Huanyu Wang, Songyuan Li, and Li Deep RGB-D saliency detection with depth- sensitive attention and automatic multi modal fusion. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 1407-1417. 2021.
Chung- Chung-Chi Tsai, Weizhi Li, Kuang-Jui Jui Hsu, Xiaoning Qian, and Yen-Yu Lin. Image co-saliency detection and co-segmentation via progressive joint optimization. IEEE Transactions on Image Processing, 28(1):56-71, 2018.
Chung-Chi Tsai, Kuang-Jui Hsu, Yen-Yu Lin, Xiaoning Qian, and Yung- Yu Chuang. Deep co-saliency detec- tion via stacked autoencoder-eeablled fusion and self-trained cnns. IEFF Transactions Multimedio, 22(4): 1016-1031, 2019.
Wei-Chih Tu, Shengfeng He, Qingxiong Yang, and Shao- Chien. Real-time salient object detection with a minimum spanning tree. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pp. 2334-2342, 2016.
Wenguan Wang, Jianbing Shen, Fang Guo, Ming-Ming Cheng, and Ali Borji. Revisiting video saliency: A large-scale henchmark and new model. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 4894-4903, 2018.
13

---

Published as a conference paper at ICLR 2022
Wenguan Wang, Qiuxia Lai, Huazhu Fu, Jianbing Shen, Haibin Ling, and Ruigang Yang. Salient object de- tection the deep learning era: An in-depth survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
The Wu, Su, and Qingming Huang. Cascaded partial decoder for fast and accurate salient object detection. In Proceedings of the IEEE Conference on Compuder Vision and Pattern Recognition, pp. 3907-3916. 2019.
Binwei Xu, Haoran Liang, Ronghua Liang, and Peng Chen. Locate globally, segment locally: A progres- sive architecture with knowledge review network for salient object detection. In Proceedings of the AAAI Conference on Artificial Intelligence. pp. 3004-300000 0044-,0122222 2021.
Kelvin Xu, Jimmy Ba, Ryan Kiros. Kyunghyun Cho, Aaron Courville. Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pp. 2048-2057, 2015.
Qingsong Yao, Quan Quan, Li Xiao, and Kevin Zhou. One-shot medical landmark detection. arXiv preprint 1.X.v:2103.04550 2021a.
Qingsong Yao. Li Xiao, Peihang Liu, and Kevin Zhou. Label-free segmentation covid-19 lesions in lung ct. IEEE Transactions on Medical Imaging, 2021b.
Dingwen Zhang, Junwei Han, Yu Zhang. Supervision by fusion: Towards unsupervised learning of deep salient object detector. In Proceedings of the IEFE International Conference on Computer Vision, pp. 4048- 4056, 2017.
Jing Zhang, Tong Zhang, Yuchao Dai, Mehrtash Harandi, and Richard Hartley. Deep unsupervised saliency detection: A multiple noisy labeling perspective. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pp. 9029 9038, 2018.
Jing Zhang, Deng-Ping Fan, Yuchao Dai, Saced Anwar, Fatemeh Sadat Saleh, Tong Zhang, and Nick Barnes. UC-Net: Uncertainty inspired RGB-D saliency detection via conditional variational autoencoders. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8582-8591, 2020a.
Jing Zhang, Xin Yu, Aixuan Li, Peipei Song, Bowen Liu, and Yuchao Dai. Weakly superv-s salient object detection via scribble annotations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12546-12555 2020b.
Miao Zhang, Jingjing Li, Wei Ji. Yongri Piao, and Huchuan Lu. Memory-oriented decoder for light field salient object detection. In Advances in Neural Information Processing Systems, pp. 898-908, 2019.
Miao Zhang, Sun Xiao Fei. Jie Liu, Shuang Xu, Yongri Piao, and Huchuan Lu. Asymmetric two-stream archi- tecture for accurate RGB-D saliency delection. In Proceedings of the European Conference on Computer Vision, 2020c.
Miao Zhang, Wei Ji, Yongri Piao. Jingjing Li, Yu Zhang, Shuang Xu, and Huchuan Lu. Lfnet: Light field fusion network for salient object detection. IEEE Transactions on Image Processing, 29:6276-622 6287, 2020d.
Miao Zhang, Weisong Ren, Yongri Piao, Zhengkun Rong, and Huchuan Lu. Select, supplement and focus for RGB-D saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3472-3481, 2020e.
Miao Zhang, Yu Zhang, Yongri Piao, Beiqi Hu, and Huchuan Lu. Feature reintegration over differential treat- ment: A top-down and adaptive fusion network for RGB-D salient object detection. In ACM International Conference on Multimedia, 2020f.
Miao Zhang, Jie Liu, Yifei Wang, Yongri Piao. Shunyu Yao, Wei Ji. Jingjing Li, Huchuan Lu, and Zhongxuan Luo. Dynamic sontext-sensitive filtering network for video salient object detection. Proceedings of the IEEE International Conference on Computer Vision, pp. 1553-1563, 2021.
Jia- Ma-Xing Zhao, Yang Cao, Deng-Ping Fan, Ming-MMng Cheng. Xuan Yi Li, and Le Zhang. Contrast prior and fluid pyramid integration for RGBD salient object detection. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 3927-3936, 2019.
Jiawei Zhao, Yifan Zhao, Jia Li, and Xiaowu Chen. Is depth really necessary for salient object detection? In ACM International Conference on Multimedia, 2020a.
Xiaoqi Zhao, Youwei Pang, Lihe Zhang, Huchuan La, and Lei Zhang, Suppress and balance: A simple gated network for salient oh In Proceedings the Furopean Conference on Computer Vision, 2020b.
14

---

Published as a conference paper at ICLR 2022
Xiaoqi Zhao, Lihe Zhang, Youwei Pang, Huchuan Lu, and Lei Zhang. A single stream network for robust and real-time RGB-D salient object detection. Proceedings of the European Conference on Computer Vision, 2020c.
Xiaoqi Zhao, Youwei Pang. Jiaxing Yang. Zhang, and Huchuan Lu. Moute-source fusion and automatic predictor selection for zero-shot video object segmentation. ACM International Conference on Multime- dia, pp. 2645-2651 2021a.
Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu, Automatic polyp segmentation via multi-scale subtraction network. In International Conference on Medical Image Computing and Cosputer-Assistd Intervention. Springer, 2021b.
Xiaoqi Zhao, Youwei Pang. Lihe Zhang, Huchuan Lu, and Xiang Ruan. Self-supervised pretraining for rgb-d salient object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Tao Zhou, Deng-Ping Ping Fan, Ming-Ming Cheng, Jianbing Shen, and Ling Shao. RGB-D salient object detection: A survey. Computational visual media, pp. 1-33, 2021.
Chunbiao Zhu, Ge Li, Xiaogiang Guo, Wenmin Wang, and Ronggang Wang. A multilayer backpropagation saliency detection algorithm based on depth mining. In International Conference Computer Analysis of Images and Patterns, pp. 14-23, 2017a.
Chunbiao Zhu, Ge Li, Wenmin Wang, and Ronggang Wang. An innovative salient object detection using center- dark channel prior. Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1509-1515 1515, 2017b.
Chunbiao Zhu, Xing Cai, Kan Huang, Thomas H Li, and Ge Li. PDNet: Prior-model guided depth-enhanced network salient object detection. IEEE International Conference on Multimedia and Expo, pp. 199- 204, 2019.
Wangjiang Zhu, Shuang Liang. Yichen Wei, and Jian Sun. Saliency optimization from robust background detection. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 28 144 2821, 2014.
15

---

Published as a conference paper at ICLR 2022
A APPENDIX
In this appendix, we first elaborate the evaluation metrics used in this paper, in Appx A.I Then, more details and experiments of our DSU are presented in Appx [A.2] including the detailed structures of D-Sal and D-NonSal CNNs, more experiments on unsupervised and tully-supervised scenarios, and other pseudo- label initialization. These empirical results consistently demonstrate the effectiveness and scalability of our approach. In A.3 we discuss the potential limitations which can he addressed the near future. In Appx [A.5] we additionally conduct some interesting experiments to further verify the efficiency of our method. Finally, in Appx A.6 we investigate several potential directions that deserve further exploration.
illustrate on the implementation details of unsupervised pipeline. In A.44
A.1 EVALUATION METRICS
We adopt four widely-used metrics in SOD to evaluate the performance saliency models, i.e., E-measure, F-measure, weighted F-measure and MAE. The lower the MAE, the better. For other metrics, the higher score is better. Concretely, F-measure is overall performance measurement and is computed by the weighted harmonic mean of the precision and recall:
FB (1 (82) Precision Recall
3 Precision Recoll
where 3 is 0.3 emphasize the precision (Achanta al. 2009). We different fixed [0, 255] thresholds to compute the F-measure metric. Weighted F-measure (F3 (Margolin et al. 2014 its weighted measurement. MAE (M) (Borji et al. (0015) represents the average absolute difterence between the predicted
saliency map and ground truth. is used to calculate how similar normalized saliency maps S E [o, 1]
is compared to the ground truth G E [0, 1] 1]WxH;
H W X EE
1
MAE
8(x, y) - ((x, x)),
(10)
where W and H denote the width and height of S, respectively. Enhanced-alignment measure (Eg) (Fan al. 2018) is proposed based on cognitive vision studies to capture image-level statistics and their local pixel
matching information, as
W
H
FE
Se (i..),
(11)
W
where (-)-) is the aahanced-alignmeen matrix Fan et al. 2018 which reflects the correlation hetween and G after subtracting their global means.
A.2 MORE ANALYSES ON THE PROPOSED DSU
Detailed structures. The detailed structures of the D-Sal CNN and D-NonSal CNN are illustrated Fig. The 'Conv 3x3' represents a convolutional layer with the kernel size of 3.
Visual results for unsupervised RGB-D SOD models. More visual comparisons with unsupervised RGB-D saliency models are shown in Fig. X where our unsupervised pipeline generates promising saliency prediction that is close to true label.
Comparison with current state-of-the-art RGB- SOD methods under suplevsue scenario. As discussed in Sec. 4.4 of the main text, our method can boost the performance of existing SOD models greatly, by applying DSU existing meth- ods. In this section, we further compare our fully- supervised variant of DSU with 25 existing state-of- the-art RGB-D D saliency models. We adopt the same feature-extraction backbone (Wu et al. (019)) in DCF (Ji et al. 2021a) to build the saliency network in the DSU framework. Numerical results in Table g and visual results in Fig. consisteety show the su- periority and scalability of our method.
Conv 3x3
BN + ReLU
Conv 3x3 3x3
BN + ReLU
Conv 3x3
BN ReLU
Conv 3x3
BN + ReLU
Conv 3
BN ReLU
D-Sal CNN
Cony 3x.3
BN + ReLU
Cony 3x3
BN + ReLU
Cony 3x3
BN + ReLU
Conv 3x3
BN + ReLU
Cony 3x3
BN + ReLU
D-NonSal CNN
Figure 7: Detailed structure of D-Sal and D-NonSal CNNs in DSU, BN means batch normalization opera- tion.
Pscudo-label Initialization. We investigate other initial pseudo-labels (e.g., DCMC (Cong et al.) 2016)), to further verify the effectiveness of our pro- posed method. The superior performance is also observed, by comparing backbone with ours: 0.167 vs. 0.146 on NJUD, 0.119 vs. 0.084 on NI.PR, 041 vs. 0.1 on STERE and O. 0.6 vs. 0.128 1828 on DUTL.FDeeth, using MAE metric.
16

---

Published as a conference paper at ICLR 2022
A.3 IMPLEMENTATION DETAILS
The proposed deep unsupervised pipeline is implemented with PyTorch and trained using single Tesla P40 GPU. Both Saliency and Depth Networks employ the same saliency feature extraction backhone (CPD (Wal al. 2019)). which is equipped with the encoder of ResNet-50 (He et al. 2016). with initial parameters pre- traind ImageNet (Krizhevsky al. 2012). All training & testing images uniformly resized the size 352 352. Throughout training, the learning rate set mini-batch size of 10. Our approach is trained in an unsupervised manner, ie., without any human annotations, where initial pscudo-labels are the outputs of handcrafted RGB-D saliency method CDCP (Zhu et al. 2017h). As for the propesed attentive training strategy, its alternation interval is set amounting to 2r 6 epochs training round. During inference, our approach directly predicts saliency maps based RGB image, without accessing any depth map.
10 and Adam optimizer used with
A.4 LIMITATION AND DISCUSSION
As demonstrated this paper. deep unsupervised RGB-D saliency detection approach achieves appealing trade-off between detection accuracy and human annotation consumption. However, due the lack of accurate pixel-level annotations, the model still fails to comprchensively detect the finegrrined details, i... the edges of salient objects. To tackle this challenge, doable solution is to introduce auxiliary edge constraint. For example, the edge detection loss can be employed to low-level features of the model, which forces the model produce discriminative features highlighting object details (Zhang al. 20206 Liu et al. 2019). The edge maps can be generated by classical Canny operator (Canny) 1986) in an unsupervised manner. Hopefully this could encourage more inspirations and contributions to this community and further pave the way for its booming future.
Image
Depth
GT
Ours
DCMC
CDCP
SE
LHM
MB
Figure 8: Qualitative comparison with unsupervised RGB-D SOD models, means ground-truth.
17

---

Published as a conference paper at ICLR 2022
Table 8: Quantitative results of fully-supervised RGB-D saliency detection methods. The best results are highlighted in boldface. When evaluating the newly released DUTLF-Dephee -epph dataset, the specific setup used by (Piao al. 2019) adopted to make fair comparison.
NJUD
NLPR
STERE F5" 1
DUTLF-Depth
Method
Fig
F5
1
F5
M
Fig
FF
Fgt
MM
Fig
1
Fg
1
M
Fg
FF"
Fg
MM.
CIME et 2017
864
.732
.788
.869
.691
.723
.056
8441
.747
.771
.086
.884
.792
.097
DF Qur MF 2011
818
552
.744
.151
8338
-244
.682
099
.61
596
.742
141
842
.542
.748
.145
2018
.866
.916
.772
066
TANeT
MANeTCChen Then & 2019
.833
812
844
.061
.916
.789
.795
041
893
804
.335
0660
866
.712
.779
.93
PDNet
798
659
MMC1
Teen et a
2019
.78
749
813
.779
.571
688
7229
059
873
757
.229
0668
8555
6.36
.753
.113
CPPP Ziiao
2019
DMRA (Paao ot m 2019
908
853
872
.051
942
845
855
031
923
841
876
0499
927
.588
883
048
SSF Zhiing
913
871
886
044
949
874
.875
026
.211
850
.66
046
946
.894
9144
034
AZdelE (1150 all
20.00
.897
.81
.874
.051
.945
.667
.878
028
.915
.85
.874
044
.924
.864
.800
.043
JL-DCY Iiu
2020
.954
.882
.878
.022
911
857
.669
.400
.931
.863
.833
0013
S2MA Lau el
20:0
.938
.522
.853
030
.907
.855
.855
.051
.921
.861
.866
.04
UCNet
2020a
.953
.878
.900
.025
.922
.667
.885
.039
.903
.821
.856
.056
FROT
A0201
.917
.862
.879
.946
.863
.688
029
.925
.888
.872
.042
.902
039
DANet
20200
.913
860
863
.047
.943
8554
857
0.0
.920
845
.855
046
847
668
.756
097
.915
.948
CMWNet
20200
.100
.555
.878
.047
.940
.856
.359
022
.917
.447
.869
004
916
X31
866
.056
.949
PGAR
hen & Tu 2020
.915
.71
.893
042
.955
.811
.885
002
.919
856
.800
.041
944
.899
.944
.035
ATSA
883
BBSNe Pall arale
20200
.244
884
.02
.035
952
.879
.882
023
925
858
885
001
833
663
.774
.120
HAINe eT
20121
.921
.877
.898
.088
.556
890
.891
024
.25
.771
.883
040
.938
.883
.006
.018
RD3D Chen ctal
2021
918
889
9000
.07
956
894
.90
.022
926
877
885
038
.499
900
.144
.031
DSA (Sun 2021
923
889
897
033
DCF
a 20216
924
893
.02
.035
.957
892
891
.021
.927
873
8835
0.39
.952
9009
.96
.030
Ours (Tully supervised)
.957
.897
.907
.895
.037
I
I
I
r
r
r
I
I
I
ft
Image Depth
GT
Ours
DCF
DSA'F
HAINet
BBSNet
S2MA
UCNet FRDT
Figure 9: Qualitative comparison of fully-supervised RGB-D SOD methods. Obviously, our fully- supervised variant infers more appealing saliency maps compared to existing SOTA models, includ- ing DCF (Ji al.) 202 DF (Sun al. 2021). HAINet (Li et 2021a). BBSNet (Fan al. 2020b), S2MA (Liu et al. 2020), UCNet (Zhang et al. 2020a) and FRDT (Zhang et al. 2020f).
18

---

Published as a conference paper at ICLR 2022
A.5 ADDITIONAL ANALYSIS
In this paper, our DSU design achieves appealing trade-off between model performance and complexity since our method has additional benefits of not introducing extra test-time cost and not relying on the depth map during the inference stage, as shown in Table [ and Fig. [ Mcanwhile, our method can he easily adapted to integrate useful depth cues by engaging the proposed DSU and DLU. In Table 2 we conduct an interesting experiment, comparing the model performance, inference speed, FI.OPs and parameter number, when using only RGB stream versus using RGB and depth simultaneously during inference. It observed that introduc- ing depth information can further improve the model performance, but at the same time, it will lead to the decrease of inference speed, and the increases of FLOPs and parameters. These results further demonstrate the effectiveness and efficiency of our DSU design.
map
Table 9:
Quantitative results of our DSU when using only RGB stream versus using RGB and depth simultaneously during inference.
Model Complexity
Model Performance (NLPR)
Speedt
FLOPs]
Parumeters.
Egt
Fg'
Fat
Mi
Our lightweight design (only using RGB)
35 FPS
17.87 G
47.85 MB
0.879
0.657
0.745
0.065
with additional depth (using RGB and depth)
21 FPS
45.16 G
96.53 MB
0.885
0,670
0.762
0.062
A.6 FUTURE DIRECTION
In this paper. we make the earliest effort to explore deep unsupervised learning in RGB-D SOD, which is demonstrated to achieve appealing performance without involving any human annotations during training. To our best knowledge, this is the first such attempt in RGB-D SOD, Mcanwhile, we also investigate several potential directions for future research as follows.
(1) Exploring on the utilization depth. The lack of explicit pixel-level supervision brings new challenge to the RGB-D SOD task, that is, inconsistency and large variations in raw depth maps. In this paper. we address it from two aspects. For depth variations between salient objects and background (i.e., possible inter-class variance), depth- eepth-disentangled network is designed to learn the discriminative saliency cues and non-salient background from raw depth map. For variance within salient objects (i.e., possible intra-object variance), we use thresholding and normalization operations to remove illegal numbers, and apply CRF to smooth the updated pseudo-labels in DLU. One promising future direction is that we can include some post-processing operations to make highlighted region on the "depth-revealed saliency map" uniform before using it to update pseudo- labels. Here we provide extended experiment in Table TQ where we smooth the inconsistent depth using a new CRF before feeding it to the DLU. This results in higher performance benefiting from more uniform depth. We believe more dedicated algorithm is worth exploring in future work to deal with inconsistent depth.
Table 10: Ablated experiment on the utilization of depth.
NLPR
NJUD
Ect
FB'
Fat
M4
Ext
F51
Fat
M1
Our DSU framework
0.879
0.657
0,745
0.065
0,797
0.597
0.719
0.135
Our DSU using saliency-guided depth with CRF
0.885
0.661
0.751
0.064
0.799
0.603
0.724
0.133
(2) Exploring on alleviating noise in unsupervised learning. Noisy problem has always been a universal and inevitable problem for unsupervised lcarning. In this paper, the proposed ATS strategy is able to alleviate this problem by properly re-weighting to reduce the influence of the ambiguous pscudo-label during training. One promising future direction is that we can design new algorithus to model the uncertainty regions and train the saliency network with partial BCE loss alleviate the noisy issue of pseudo-labels.
(3) Exploring on generating high-quality pseudo-labels. In our paper, a (DSU) framework is designed to automatically produce pseudo-labele with iterative follow-up refinements, which is able to provide more trustworthy supervision signals for training the saliency network. For future work, we can explore other algorithms to generate high-quality pseudo-labels, for example, 1) imposing contrastive loss (e.g., using superpixel candidates with ROI pooling construct contrastive pairs) enlarge the feature distance between foreground and background; 2) involving partially labeled scribbles to help generate more accurate pseudo-labels.
saliency update
(4) Exploring on more fine-grained disentangled framework. In this paper, we design a depth disentangled saliency update framework to decompose depth information into saliency-guided and non-saliency-gune guided depths to promote saliency, motivated by the nature of the class-agnostic hinary salient object segmentation
19

---

Published as a conference paper at ICLR 2022
lask. It is worthy exploring in the future design more fine-grained disentangled framework. For example, in addition the salieney-guides depth and saliency-guides depth, we can additionally model the 'not sure regions by learning an uncertainty map (using well-designed new algorithms such as Monte Carlo Dropout (Gal) & Ghahramani] 2016). The learned uncertainty map, which reveals the 'not sure' regions, can engage in the pseudo-label training process to mitigale the noises of uncertain pseudo-labels, using partial BCE loss.
(5) Exploring
applications. SOD useful for variety of downstream applications including image classification (Chen al. 2021c| et al. 2021 2022 [Nng al.
c light- field data (Zhang et all 2021b 2020al Yao et al. a Ning al. 2021b 2020 and video analysis (Zhang et al. 2021 Zhao et al. 102 etc. This is benefiting from its generic mechanism highlight class-agnostic objects
2019 2020d), medical image processing (Zhao et al. 2021b Ji et al.
scene. For unsupervised RGB-D SOD, it can be regarded pre-task prior effectively improve the performance of target task (e.g., weakly supervised semantic segmentation, action recognization), and avoid more laborious efforts.
As discussed above, we summarize five potential research directions for deep unsupervised RGB-D D salient object detection. Hopefully this could encourage more inspirations and contributions to this community.
20


Reviews:
[{"review_id": "Sc40ZMp9uHE", "reviewer": "Reviewer_WhBE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written and easy to understand.\nThis paper pointed out the two important issues of unsupervised RGB-D saliency detection; accurate pseudo-label generation and noisy label handling method.\nThe paper proposed an effective framework to solve the issues and the proposed method consists of two main components; Depth-disentangled Saliency Update (DSU) framework and Attentive training strategy (ATS).\nDSU framework generates and refines accurate pseudo-label by utilizing depth map disentanglement and iterative label refinement.\nThe depth map disentanglement and iterative label refinement complementarily improve each other's quality.\nATS reduces the influence of ambiguous pseudo-label and concentrates on the more reliable training examples by adaptive reweighting strategy.\nThe contributions of the proposed method are novel. The experimental results including qualitative and quantitative results are impressive.\nThe extensive ablation study and detailed analysis support the proposed method's effectiveness.\n\n[Minor issue]\n1. The description of \"Backbone\" in the legend of figure 4 is not described in sec 3.3 Attentive training strategy. \nIt seems the \"backbone\" is identical with the \"backbone\" of sec 4.3 Analysis of the empirical result.\nI recommend the author to add a description or point another description.\n", "summary_of_the_paper": "This paper proposed an effective framework for the unsupervised RGB-D saliency detection task.\nThe proposed framework consists of two key components. One is the depth-disentangled saliency update framework that is iteratively refined & update the pseudo-labels. The other one is an attentive training strategy that alleviates the influence of noisy pseudo-labels.\nThe two components mainly tackle the important issues of deep unsupervised RGB-D saliency detection; accurate psuedo-label generation and noisy pseudo-label handling.\nBased on the proposed components, the proposed framework achieves state-of-the-art results.\n", "summary_of_the_review": "This paper clearly states the problems of unsupervised RGB-D saliency detection and proposed appropriate solutions.\nThe contribution and novelty are clear. Extensive ablation studiesy reveal the effectiveness of its components.\nBased on proposed solutions, the proposed network achieves state-of-the-art results.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OQWweIHsK2Z", "reviewer": "Reviewer_kVSL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The idea of decomposing depth cues into saliency-guided depth and non-saliency-guided is novel and makes a good sense, leading to the proposed network architecture and the design of the training loss.\n2. The evaluation is extensive and well designed ablation studies are provided.\n3. The paper is well organised.\n\nWeaknesses:\n\n1. The core issue is whether the work is an 'unsupervised' method or not given that this is an important claim made in the paper. In my opinion, this work is essentially a weakly supervised method for RGB SOD where the depth information is used as additional knowledge to improve the intial labels generated by the handcrafted method. I also notice that in Table 1, the proposed method is compared with USD and DeepUSPS, which are both unsupervised RGB SOD methods without any involvement of depth information. To this end, the comparison is barely fair as it is well-known that depth information can significantly improve SOD for the methods based only on RGB information.\n\n2. It is stated at the end of Section 3.2 that in the inference stage, \"only RGB images are used for predicting saliency\". I wonder if the learned mechanism which uses the depth information to refine the RGB-based pesudo label is generalised well on the images not from the same dataset. It might be worthy evaluating the method by training the DSU network on one dataset and then testing it on another one.\n\n3. Despite that the DSU network, particularly the depth-disentangling scheme, offers an effective way to exploit depth information for the improvement of RGB SOD, there is a lack of insightful analysis for the thorough understanding of the scheme. For instance, if we extend the \"saliency-guided and non-saliendy-guided\" binary taxonomy to a ternary one, would the DSU network performs better? Note that this won't slow down the detection in the inference. ", "summary_of_the_paper": "The paper presents an unsupervised method for RGBD SOD. It internally generates pseudo labels for training by extracting and refining/updating handcrafted features under the framework of ‘depth-disentangling’. Since the updating module can be removed during the inference, it achieves high efficiency.  The method also demonstrates SOTA performance on several benchmarks. ", "summary_of_the_review": "Based on the strengths and weaknesses listed above, I am slightly negative to the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Apptn1Gkub", "reviewer": "Reviewer_xcAY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is presented clearly, and the experiments are comprehensive. Additionally, the idea of jointly deriving the saliency training labels from the depth and RGB data is good. However, I have a few questions listed below:\n\n1.  Why discard the available depth signal during inference? This paper discusses RGBD saliency; however, the proposed method is still under the RGB saliency detection framework. The idea will be more impactful if a gated scheme fuses depth information to promote saliency whenever available. Otherwise, the experimental datasets you chose are RGBD, and you only use RGB & Depth to generate the training labels and discard the valid depth signal during inference. Is saving compute the primary reason? \n\n2. When salient objects contain different levels of depth (saying a bus in a frontal view), will the depth distilled saliency/non-saliency signal remain helpful to the pseudo-labels? My concerns come from the observation that D_{Sal} & D_{NonSal} looks similar to the depth map, so when there is a long object with a wide range of depth, the value among the object area in the D_{Sal} won't be uniform; further, if we element-wise multiply such map with the Sal_{pred} in the DLU, will it just make the result even worse?\n\n3. Does error propagation exist in your proposed iterative optimization scheme? What if the depth disentangled saliency signal becomes unreliable, and the iterative optimization scheme guides the network to learn in the wrong direction?   \n\n4. When comparing with the handcrafted UnSOD in Table1, do you retrain those methods on your training data to ensure a fair comparison?\n\n5. Inside ATS, given ten epochs, what epoch use step one, and what epoch use step two? Also, is there any stop mechanism besides the current heuristic one?\n\n6. The idea of utilizing the handcrafted saliency method as unsupervised training data and using auxiliary information to refine such training data for the more trustworthy signal is also presented in the following papers:\n(1) \"Deep co-saliency detection via stacked autoencoder-enabled fusion and self-trained cnns.\"\n(2)\"Unsupervised CNN-based co-saliency detection with graphical optimization.\" \nHowever, the difference separating with them is not discussed in the paper.", "summary_of_the_paper": "This paper presents an unsupervised RGB saliency detection model to learn from the pseudo-labels given by the traditional handcrafted approach. To enhance the quality of the pseudo labels, the author proposes to disentangle the depth data to promote the saliency signal and compress undesired noise. Last but not least, an attentive training strategy is integrated to teach the network from iteratively updating supervisory labels. Experimental results demonstrate the effectiveness of the proposed depth-disentangled saliency update framework and can help improve the performance of on-the-shelf approaches.", "summary_of_the_review": "Overall, I think it is currently a borderline paper since the idea is good, which can benefit researchers in related fields. Meanwhile, this work is presented well, and the experiments are comprehensive. However, I hope to check the answer to my questions before making further judgments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received two accepts and 1 marginally above acceptance recommendations. The authors provided satisfactory answers, mostly on clarifying the unsupervised learning methodology, in conjunction with the MAA recommendation. I recommend the paper be accepted as poster.
Decision: Accept (Poster)