DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation
Authors: Bowen Yin
Xuying Zhang
Zhong-Yu Li
Li Liu
Ming-Ming Cheng
Qibin Hou
Venue: ICLR 2024 poster (2024)
Keywords: RGB-D Semantic Segmentation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose DFormer, an innovative RGB-D pretraining framework for enhancing RGB-D segmentation tasks, achieving new state-of-the-art performance with less than half the computational cost of current best methods.

Abstract: We present DFormer, a novel RGB-D pretraining framework to learn transferable representations for RGB-D segmentation tasks. DFormer has two new key innovations: 1) Unlike previous works that encode RGB-D information with RGB pretrained backbone, we pretrain the backbone using image-depth pairs from ImageNet-1K, and thus the DFormer is endowed with the capacity to encode RGB-D representations; 2) DFormer comprises a sequence of RGB-D blocks, which are tailored for encoding both RGB and depth information through a novel building block design. DFormer avoids the mismatched encoding of the 3D geometry relationships in depth maps by RGB pretrained backbones, which widely lies in existing methods but has not been resolved. We finetune the pretrained DFormer on two popular RGB-D tasks, i.e., RGB-D semantic segmentation and RGB-D salient object detection, with a lightweight decoder head. Experimental results show that our DFormer achieves new state-of-the-art performance on these two tasks with less than half of the computational cost of the current best methods on two RGB-D semantic segmentation datasets and five RGB-D salient object detection datasets. Code will be made publicly available.

Published as a conference paper at ICLR 2024
DFORMER: RETHINKING RGBD REPRESENTATION LEARNING FOR SEMANTIC SEGMENTATION
Bowen Yin Xuying Zhang' Zhongyu Li!
Li
Ming-Ming Cheng!
Qibin Hou
VCIP, School of Computer Science, Nankai University
2 National University of Defense Technology
3 Nankai International Advanced Research Institute (Shenzhen Futian) bowenyinemail nankai edu andrewhoux@gmall-o
ABSTRACT
We present DFormer, novel RGB-D pretraining framework learn transferable representations for RGB-D segmentation tasks. DFormer has two new key inno- vations: Unlike previous works that encode RGB-D information with RGB pretrained backbone, we pretrain the backbone using image-depth pairs from ImageNet-1K, and hence the DFormer is endowed with the capacity to encode RGB-D representations; 2) DFormer comprises sequence of RGB-D blocks, which are tailored for encoding both RGB and depth information through a novel building block design. DFormer avoids the mismatched encoding of the 3D geom- etry relationships in depth maps by RGB pretrained backbones, which widely lies in existing methods but has not been resolved. We finetune the pretrained DFormer on two popular RGB-D tasks, i.e., RGB-D semantic segmentation and RGB-D salient object detection, with a lightweight decoder head. Experimental results show that our DFormer achieves new state-of-the-arr performance on these two tasks with less than half of the computational cost of the current best methods two RGB-D semantic segmentation datasets and five RGB-D salient object detec- tion datasets. Our code available at: https://github.. www.oo///CII///DDD/OO//o/ FGDDrDrrrmer..
INTRODUCTION
Hankbone
RGB
RGR Backbone
Finotuiing
Cat
Internction
Depth
RGB
RGB Pretraining
DGB
RGB-
Findtuning
RGB-D
Cal
RGB-D Pretraining (Ours)
Comparisons between the existing popular training pipeline and for RGB-D - segmentation. RGB pretraining: Recent mainstream methods adopt two RGB pretrained backbones to separately encode RGB and depth information and fuse them at each stage. RGB-D pretraining: The RGB-D backbone in DFormer learns transferable RGB-D representations during pretraining and then is finetuned for segmentation.
Figure 1:
With the widespread use of 3D sensors, RGB-D data is becoming increasingly available to access. By incorporating 3D geometric information, it would be easier to distinguish instances and context, facilitating the RGB-D research for high-level scene understanding. Mcanwhile, RGB-D data also presents considerable potential in large number of applications, e.g., SLAM (Wang et al., 2023), automatic driving (Huang et al., 2022), and robotics (Marchal et al., 2020). Therefore, RGB-D research has attracted great attention over the past few years.
"Qibin Hou the corresponding author.

---

Published as a conference paper at ICLR 2024
Fig. (top) shows the pipeline of current mainstream RGB-D methods. As can be observed, the fea- tures of the RGB images and depth maps are respectively extracted from two individual RGB pre- trained backbones. The interactions between the information of these two modalities are performed during this process. Although the existing methods (Wang et al., 2022; Zhang et al., 2023b) have achieved excellent performance ignored: i) The backbones in the RGB-D tasks take an image-depth pair as input, which is incon- sistent with the input of image in RGB pretraining, causing a huge representation distribution shift; ii) The interactions are densely performed between the RGB branch and depth branch during finetuning. which may destroy the representation distribution within the pretrained RGB backbones; iii) The dual backbones in RGB-D networks bring more computational cost compared to standard RGB methods, which is not efficient. We argue that an important reason leading to these issues the pretraining manner. The depth information is not considered during pretraining.
several benchmark datasets, there are three issues that cannot be
Taking the above issues into account, a straight- forward question arises: Is it possible to specif- ically design an RGB-D pretraining framework to eliminate this gap? This motivates us to present a novel RGB-D pretraining framework, termed DFormer, shown in Fig. (bottomm). During pretraining, we consider taking image- depth pairs', not just RGB images, as input and propose to build interactions between RGB and depth features within the building blocks of the encoder. Therefore, the inconsistency between the inputs of pretraining and finetuning can be naturally avoided. In addition, during pretrain- ing, the RGB and depth features can efficiently interact with each other in each building block, avoiding the heavy interaction modules outside the backbones, which is mostly adopted in cur- rent dominant methods. Furthermore, we also observe that the depth information only needs a small portion of channels to encode. There no need to use whole RGB pretrained backbone extract depth features as done in previous works. As the interaction starts from the pretraining stage, the interaction efficiency can be largely improved compared to previous works as shown Fig. 2.
CMNext
57
Ours
MuliMAE
56
55
TokenFusion
mloU
54
Omuzivare
53
Model
mtoll
Parum
Flogs
Latency
52
CMX B2
54.4
86.6M
67.60
37.5ms
51
OursB
35.6
29.5M
41.96
35.0ms
CMNNect-4
56,9
119.6M
131,93
78:2ms
50
OursLL
57.2
38.0M
65.76
35. 7m.7
0
50
100
150
200
250
FLOPs/G
Figure 2: Performance vs. computational cost on the NYUDepthy2 dataset (Silberman et al., 2012). DFormer achieves the state-of-the-ar 57.2% mloU and the best trade-off compared to other methods.
We demonstrate the effectiveness of DFormer on two popular RGB-D downstream tasks, i.e., se- mantic segmentation and salient object detection. By adding a lightweight decoder on top of our pretrained RGB-D backbone, DFormer sets new state-of-the-art records with less computational cost compared to previous methods. Remarkably, our largest model, DFormer-L, achieves a new state- of-the-art result, i.e., 57.2% mIoU on NYU Depthv2 with less than half of the computations of the second-best method CMNext (Zhang et al., 2023b). Meanwhile, our lightweight model DFormer- T is able to achieve 51.8% mloU on NYU Depthv2 with only 6.0M parameters and 11.8G Flops. Compared to other recent models, our approach achieves the best trade-off between segmentation performance and computations.
To sum up, our main contributions can be summarized as follows:
We present a novel RGB-D pretraining framework, termed DFormer, with a new interac- tion method to fuse the RGB and depth features to provide transferable representations for RGB-D downstream tasks.
We find that in our framework it is enough to use small portion of channels to encode the depth information compared to RGB features, an effective way to reduce model size.
Our DFormer achieves new state-of-the-an performance with less than half of compu- tational cost of the current best methods two RGB-D segmentation datasets and five RGB-D salient object detection datasets.
Forr depth maps, employ widely used depth estimation model (Bhat et al., 2021) to predict depth map for each RGB image, which found works well.
2

---

Published as a conference paper at ICLR 2024
RGBB SteeSt
Stage
Stage 2
Stage 3
Stage 4
1
RGB D Block
H H4 W/4
RGB- D Block
H8t W/8
H/16x W/11
RGB- WGB--BBookk Block
H/32x W/32
Connatenation
Deroder Head
H/Bx W/B
Depti
D> Sbam
Layer
RGB-
Prediction
xN
XN2
xNa
XN4
Figure 3: Overall architecture of the proposed DFormer. First, we use the pretrained DFormer to encode
the RGB-D data. Then. the features from the last three stages are concatenated and delivered to lightweight decoder head for final prediction. Note that only the RGB features from the encoder used in the decoder.
PROPOSED DFORMER
Fig. 3 illustrates the overall architecture of our DFormer, which follows the popular encoder-decoder framework. Given an RGB image and the corresponding depth map with spatial size x W, they are first separately processed by two parallel stem layers consisting of two convolutions with kernel size 3 x 3 and stride 2. Then, the RGB features and depth features are fed into the hierarchical encoder encode multi-scale features (1/4, 1/8 1/16, 1 1/32] of the original image resolution. Next, we pretrain this encoder using the image-depth pairs from ImageNet-IK IK using the classification objective generate the transferable RGB-D representations. Finally, we send the visual features from the pretrained RGB-D encoder to the decoder to produce predictions, e.g., segmentation maps with a spatial size of H W. In the rest of this section, we will describe the encoder, RGB-D pretraining framework, and task-specific decoder in detail.
2.1 HIERARCHICAL ENCODER
As shown in Fig. 3, our hierarchical encoder is composed of four stages, which are utilized to gen- erate multi-scale RGB-D D features. Each stage contains stack of RGB-D blocks. Two convolutions with kernel size and stride are used to down-sample RGB and depth features, respectively, between two consecutive stages.
Building Block. Our building block is mainly composed of the global awareness attention (GAA) module and the local enhancement at- tention (LEA) module and builds interaction between the RGB and depth modalities. GAA incorporates depth information and aims to en- hance the capability of object localization from global perspective, while LEA adopts large- kernel convolution to capture the local clues from the depth features, which can refine the details of the RGB representations. The details of the interaction modules are shown in Fig. 4.
hxw x Croe
our
Nost
Hadamard Product
DXW X
Concatonation
Proj.
Proj
GAA
LEA
Glocal Awareness Attention
DWCCnv
Bxwx X
Wg
Wg. Wy
Concat Pool
Wg
WA
Our GAA fuses depth and RGB features to build relationships across the whole scene, en- hancing 3D awareness and further helping cap- ture semantic objects. Different from the self- attention mechanism (Vaswani al., 2017) that introduces quadratic computation growth as the pixels or tokens increase, the Query (Q) in GAA down-sampled to a fixed size and hence the computational complexity can be reduced. Tab. illustrates that fusing depth features with Q is adequate and there is no need to combine them with K V. which brings computation increment but no performance improvement. So, Q comes from the concatenation of the RGB features and depth features, while key (K) and value (V) are extracted from RGB features. Given the RGB features X. and depth features X the above process can be formulated as:
A N.W. x
X,50
x
X
x
Figure 4: Diagrammatic details how conduct in- teractions between RGB and depth features.
Q =
xx K Linear (X,9b) V Linear (XT=b).
where - denotes the concatenation operation along the channel dimension, Poolax performs adaptively average pooling operation across the spatial dimensions x k size, and Linear is linear
3

---

Published as a conference paper at ICLR 2024
transformation. Based the generated Q E R K E Rxxwxx", and V R where and w are the height and width of features in the current stage, we formulate the GAA as follows:
K 1),
Xaaa = UP(VU Soflmaxx
where (-(-) is bilinear upsampling operation that converts the spatial size from l: x k to h x In practical use, Eqn. 2 can also be extended to a multi-head version, as done in the original self- attention (Vaswani al., 2017), augment the feature representations.
We also design the LEA module to capture more local details, which can be regarded as a supplement to the GAA module. Unlike most previous works that addition and concatenation to fuse the RGB features and depth features. We conduct a depth-wise convolution with a large kernel on the depth features and use the resulting features as attention weights reweigh the RGB features via simple Hadamard product inspired by (Hou et al., 2022). This is reasonable in that adjacent pixels with similar depth values often belong to the same object and the 3D geometry information thereby can be easily embedded into the RGB features. To be specific, the calculation process of LEA can be defined as follows:
XLEA DConv (Lincar @ Linear(X?"").
(3)
where DConvexk & is depth-wise convolution with kernel size k k and + is the Hadamard product.
To preserve the diverse appearance information, we also build a base module to transform the RGB features X to X Base, which has the same spatial size as XGAA and XLEA. The calculation process of X Base can be defined as follows:
X Base DConvE (Linear(X;) (x(9 + Linear(X;") (x.%))
(4)
Finally, the features, ie., Xasa E X XLEA E RRxxw,CC X X E RR.xw, x0," are fused together by concatenation and linear projection to update the RGB features X and depth features X
Overall Architecture. We empirically observe that encoding depth features requires fewer param- eters compared to the RGB ones due to their less semantic information, which verified Fig. and illustrated in detail in the experimental part. To reduce model complexity in RGB-D block, we use a small portion of channels to encode the depth information. Based on the configurations of the RGB-D blocks in each stage, we design series of DFormer encoder variants, termed DFormer- T, DFormer-S, DFormer-B, and DFormer-L, respectively, with the same architecture but different model sizes. D is lightweight encoder for fast inference, while DFormer-L is the largest one for attaining better performance. For detailed configurations, readers can refer to Tab. 16.
2.2 RGB-D PRETRAINING
The purpose of RGB-D pretraining is to endow the backbone with the ability to achieve the interac- tion between RGB and depth modalities and generate transferable representations with rich semantic and spatial information. To this end, we first apply a depth estimator, e.g., Adabin (Bhat et al., 2021), on the ImageNet- 1K dataset (Russakovsky et al., 2015) to generate a large number of ieage-depth pairs. Then, we add classification head on the top of the RGB-D) encoder to build the classifica- tion network for pretraining. Particularly, the RGB features from the last stage are flattened along the spatial dimension and fed into the classifier head. The standard cross-entropy loss is employed as our optimization objective, and the network is pretrained on RGB-D data for 300 epochs, like ConvNeXt (Liu et al., 2022). Following previous works (Liu et al., 2022; Guo et al., 2022b), the Adam W optimizer (Loshchilov employed and the batch size is set to 1024. More specific settings for each variant of DFormer described in the appendix.
Hutter, 2019) with learning rate le-3 and weight decay 5e-2 is
2.3 TASK-IPECIFIC DECODER.
For the applications of our DForme to downstream tasks, we just add a lightweight decoder on the top of the pretrained RGB-D backbone to build the task-specific network. After being finetuned on
4

---

Published as a conference paper at ICLR 2024
Table 1: Results on NYU Depth V2 (Silberman et al., 2012) and SUN- LUN-RGBD (Song et al., 2015), Some methods do not report the results or settings on the SUN-RGBD datasets, so we reproduce them with the same training config. indicates our implemented results. All the backbones are pre-trained on ImageNet-IR
Model
Backbone
Params
NYUDepth22
SUN-RGBD
Code
Input size
Flops
mioU
Inpul size
Flups
mloU
ACNelp (Ilu a
ResNei-50
480 640
126.70 76
530 730
163.9G
48.1
Link
SGNetao (Chen et al.)
ResNet-101
64.7M
480 x 640
108.5G
51.1
530
X 730
151 5G
48.6
Link
SA-Gatezo (Chen ot al.)
ReaNet-101
110.9M
180 6-00
193.7G
52.4
530
730
250.1G
49.4
Link
CEN2o (Wang et al.)
ResNet-101
118.2M
480 640
618.7G
530
730
790.3G
Link
CENao (Wang et al.)
ResNet-152
133.9M
180 x 640
664.4G
52.5
530
X 730
849.7G
51.1
Link
ShapeConvv3 (Cao et al.)
ResNext 101
86.8M
480 x 640
124.6G
51.3
5,30
730
161.80
48.6
Link
ES ANet2a (Seichter et al.)
ResNet-34
31.2M
480 640
34,9G
50.3
480
640
34.9G
48.2
Link
FRNetaz (2hou et al.)
ResNet-34
85.5M
180 640
115.60
53.6
530
730
150.0G
51.8
Link
PGDENe (Zhou et al.)
R4
100.7M
480 x 640
178.8G
53.7
530
X 730
229. IG
$1.0
Link
AMgg (Seichter et all)
R4
46.9M
480 640
45,4G
51.0
530
730
58.6G
48.4
Tink
Tokionasio (Wang et all)
MiT-B2
26.0M
480 640
55,2G
53.3
630
730
71.1G
50,31
Link
TokenFusionzz (Wang ot al.)
MiT-B3
45,9M
180 640
94.4G
54.2
530 X 730
122.1G
51.0*
Link
MuluMAEs2 (Bachmann et al.)
VIT-B
95.2M
640 640
267.9G
56.0
640 640
267.9K
51.1
Link
Omaivoreza (Girdhar et al.)
Swin T
19.1M
480 640
32.7G
49.7
530 730
Link
Omnivoreza (Girdhar al)
Swin-S
51.3M
480 640
59,8G
52.7
530
730
Link
Omaivorega (Girdhat et al.)
Swin-B
95.7M
180 640
109.3G
54.0
530
X 730
Link
CMXz2 (Zhung el al.)
B2
66.6M
480 X 640
67.6G
54.4
530
730
86.3G
49.7
Link
2MX (Zhang al.)
MiT-B4
139.9M
480 640
134.3G
56.3
530
730
173.8G
52.1
Link
CMX22 (Zhang et al.))
M5T.B5
1M
480 640
167.8G
530
730
217.6G
52.4
Link
CMNextas (Zhang
MiT-B4
119.6M
480
640
131.9G
56.9
530
730
170.3G
51.91
Link
DFommer- T
Ours-T T
6.0M
480 640
1
51.8
530
730
15.1G
48.8
Link
DForee-S
Owrs-S
18.7M
480
640
25.6G
53.6
630
X 730
33.0G
50.0
Link
DFormer-B
Ours-B
29.5M
180 x 6:00
41.9G
55.6
530
X 730
$4.10
$1.2
Link
DFomer L.
Ours L
39.0M
480
640
65.7G
57.2
530
730
83.3G
52.5
Link
corresponding benchmark datasets, the task-specific network is able to generate great predictions, without using extra designs like fusion modules (Chen et al., 2020a; Zhang et al., 2023a).
Take RGB-D semantic segmentation as example. Following SegNeXt (Guo al., 2022a), adopt a lightweight Hamburger head (Geng et al., 2021) to aggregate the multi-scale RGB features from the last three stages of our pretrained encoder. Note that, our decoder only uses the Xreb features, while other methods (Zhang et al., 2023a: Wang et al., 2022; Zhang et al., 2023b) mostly design modules that fuse both modalities features X and X for final predictions. We will show in our experiments that our X features can efficiently extract the 3D geometry clues from the depth modality thanks to our powerful RGB-D pretrained encoder. Delivering the depth features x to the decoder is not necessary.
3 EXPERIMENTS
3.1 RGB-D SEMANTIC SEGMENTATION
Datasets & implementation details. Following the common experiment settings of RGB-D seman- tic segmentation methods (Xie et al., 2021; Guo et al., 2022a), we finetune and evaluate our DFormer on two widely used datasets, i.e., NYUDepthv2 (Silberman et al., 2012) and SUN-RGBD (Song al., 2015). During finetuning, we only adopt two common data augmentation strategies, i.e, ran- dom horizontal flipping and random scaling (from 0.5 to 1.75). The training images are cropped and resized to 480 x 640 and 480 x 480 respectively for NYU Depthv2 and SUN-RGBD benchmarks. Cross-entropy loss is utilized as the optimization objective. We use Adam (Kingma & Ba, 2015) as our optimizer with an initial learning rate of 6e-5 and the poly decay schedule. Weight decay is set to le-2. During test, we employ mean Intersection over Union (mloU), which is averaged across semantic categories, as the primary evaluation metric to measure the segmentation performance. Following recent works (Zhang et al., 2023a; Wang et al., 2022; Zhang et al., 2023b), we adopt multi-scale (MS) flip inference strategies with scales (0.5, 0.75, 1, 1.25, 1.5).
Comparisons with state-of-the-art methods. We compare our DFormer with 13 recent RGB-D se- mantic segmentation methods on the NYUDepthv2 (Silberman et al., 2012) and SUN-RGBD (Song et al., 2015) datasets. These methods are chosen according to three criteria: a) recently published, b) representative, and c) with open-source code. As shown in Tab. 1, our DFormer achieves new state-of-the- thr performance across these two benchmark datasets. We also plot the performance- efficiency curves of different methods on the validation set of the NYUDepthv2 (Silberman et al., 2012) dataset in Fig. 2. It is clear that DFormer achieves much better performance and computation
5

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative comparisons on RGB-D D SOD benchmarks. The best results are highlighted.
Datasel
Param
Flops (G)
DES(13))
NLPR(300)
NJU2K(300)
STERE(
(.000)
SUP9229)
Metric
(M)
M
F
M
M
8
8
BBSNeta: (Zhai et al)
49.8
31.3
021
942
934
955
023
.927
9.30
953
035
931
920
941
041
.19
908
.911
055
.002
.879
.910
DCF,, (li et al)
108.5
$4.3
024
910
905
911
022
918
921
958
036
922
912
946
039
.911
902
940
052
.899
8876
.916
DSA2F2 (Sun al)
36.5
172.7
021
889
920
962
024
.897
918
950
039
901
903
923
036
.898
904
.93
CMINetgi (Zhong et al)
016
944
940
975
020
931
932
959
028
940
929
954
032
925
918
996
040
993 898 994
DSNeta1 (Wen et al)
172.4
141.2
999
956
924
.925
926
034
921
036
914
052
.86
UTANelo (2hr all)
48.6
27.4
022 921 900
952
020
9928
932
964
037
915
902
945
013
921
910
94K
048
897
873 925
BIANetg: (Zhang al)
19.6
59.9
0117
948
912
972
022
.926
928
.977
034
932
923
945
1038
.916
908
.955
016
998 889 .922
SPNetss (2hou
150.3
9417
894
VSTaI (Liu at al))
83.3
31.0
.07
910 913
976
021
.920
932
962
035
920
922
.95
0.38
.907
913
.951
010
91.5 901 .911
RD3D+22 (Chen et al))
28.9
43.3
017
946
950
.982
0122
921
933
964
033
9988
955
A137
9905
914
996
046
900
899 9228
BPGNelaz (Yang
84.3
138.6
020
932
957
973
.02
.144
.27
.959
.034
9926
.93
995
[H40
.044
.907
.944
C2DPNetog (Phang al.)
47.5
21.0
0220
937
922
948
4221
9926
928
956
4138
9111 902
.938
055
894
7782 .111
MVSalNelgz (Zhou
912
0136
SPSN22 (Lecet al.)
37.0
100.3
017
942 937
973
.02
917
923
956
032
9227
918
9449
.035
9009
9006
.941
(143
.910
8891 .932
HiDANetgs (Waet
130.6
71.5
.013 952 946
980
A02
.929
993
.611
029
939
926
954
.035
.921
.911
.966
[043
.919
/892 .927
DFormer-
5.9
947
.911
927
033
915
003
DFormer-S
18.5
10.1
.16 950
939
970
020
937
936
965
076
994
931
966
4131
928
920
951
(44
.921
898
9931
DFormer-B
29.3
16.7
.013
957
948
982
019
993
936
965
025
941
933
960
022
9911
925
.91
035
.92 908
943
DFarmes-I.
38.8
26-2
956
002
(130
923
(832 915
Table 3: RGB-D pretraining. 'RGB+RGB' pre- training replaces depth maps with RGB images during pretraining. Input channel of the stem layer is modified from to 3. The depth map is duplicated three times during finetuning.
Table 4: Different inputs of the decoder head for DFomer--. "X"9"+X means simultane- ously uses RGB and depth features. Specifi- cally, both features from the last three stages are used as the input of the decoder head.
pretrain
Finetune
mIoU (%)
Decoder input
#Params
FLOPs
mIoU (%)
RGB+RGB RGB+D (Ours)
RGB+D
$3.3
X (Ours)
29.5
41.9
55.6
RGB+D
55.6
X + X
30.8
44.8
55.5
trade-off compared to other methods. Particularly, DFormer-L. yields 57.2% mloU with 39.0M pa- rameters and 65.7G Flops, while the recent best RGB-D semantic segmentation method, i.e., CMX (MiT-B2), only achieves 54.4% mloU using 66.6M parameters and 67.6G Flops. It is noteworthy that our DFormer-B B can outperform CMX (MIT-B2) by 1.2% mloU with half of the parameters (29.5M, 41.9G vs 66.6M. 67.6G). Moreover, the qualitative comparisons between the semantic seg- mentation results of our DFormer and CMNext (Zhang et al., 2023b) in Fig. 14 of the appendix fur- ther demonstrate the advantage of our method. In addition, the experiments on SUN-RGBD (Song et al., 2015) also present similar advantages of our DForer over other methods. These consistent improvements indicate that our RGB-D backbone can more efficiently build interactions between the RGB and depth features, and hence yields better performance with even lower computational cost.
3.2 RGB-D SALIENT OBJECT DETCTION
Datasets & implementation details. We finetune and test DFormer on five popular RGB-D salient object detection datasets. The finetuning dataset consists of 2,195 samples, where 1,485 are from NJU2K-train (Ju et al., 2014) and the other 700 samples are from NLPR-train (Peng et al., 2014). The model is evaluated on five datasets, i.e., DES (Cheng et al., 2014) (135). NLPR-test (Peng et al., 2014) (300), NJUt (Ju al., 2014) (500), STERE (Niu al., 2012) (1,000), and SIP (Fan et al., 2020) (929). For performance evaluation, we adopt four golden metrics of this task, i.e., Structure-measue measure (S) (Fan et al., 2017), mean absolute error (M) (Perazzi et al., 2012), max F-measure (F) (Margolin et al., 2014), and max E-measure (E) (Fan et al., 2018).
Comparisons with state-of-the-art methods. We compare our DFormer with 11 recent RGB-D salient object detection methods on the five popular test datasets. As shown in Tab. 2, our DFomme is able to surpass all competitors with the least computational cost. More importantly, our DFormer- T yields comparable performance the recent state-of-tt of-the--rttrrr- art method SPNet (Zhou et al., 2021a) with less than 10% computational cost (5. 9M, 4.5G vs 150.3M, 68. The significant improvement further illustrates the strong performance of DFormer.

---

Published as a conference paper at ICLR 2024
Image
Depth Map
X gb
X
Xrgb
Xd
out
Figure 5: Visualizations of the feature maps around the last RGB-D block of the first stage.
Table 5: Ablation results on the components of
Table 6:
Ablations on GAA in the DFormer-S. k' means the adaptive pooling size of Q.
the RGB-D block in DFormer-S.
Model
#Params
FLOPs
mIoU (%)
Kernel Size #Params
FLOPs
mloU (%)
DFormer-S
18.7M
25.6G
53.6
3x3
18.7M
22.0G
52.7
w/o Base
14.6M
19.6G
52.1
5x5
18.7M
23.3G
53.1
w/o GAA
16.5M
21.7G
52.3
7x7
18.7M
25.6G
53.6
w/o LEA 16.3M
23.3G
52.6
9x9 18.7M
28.8G
53.6
Table 7: Different fusion methods for Q, K, and
Different fusion manners in the LEA module. Fusion manner refers to the operation to fuse the RGB and depth information in LEA.
Table 8:
V. We can see that there no need to fuse RGB and depth features for K and V.
Fusion variable #Params FLOPs mloU (%)
Fusion manner
#Params
FLOPs
mloU(% (%)
None 18.5M 18.7M
25.4G
53.2
Addition
1866M
25.1G
53.1
only Q (ours) Q, K, V
25.6G
53.6
Concatenation
19.0M
26.4G
53.3
19.3M
28.5G
53.6
Hadamard
18.7M
25.6G
53.6
3.3 ABLATION STUDY AND ANALYSIS
We investigate the effectiveness of each component here. All experiments here are conducted under the RGB-D segmentation setting on NYU Depth (Silberman et al., 2012).
RGB-D vs. RGB pretraining, To explain the necessity of the RGB-D pretraining, we attempt to replace the depth maps with RGB images during pretraining, dubbed as RGB pretraining. To be specific, RGB pretraining modifies the input channel of the depth stem layer from 1 to 3. Note that for the finetuning setting, the modalities of the input data and the model structure are the same. As shown in Tab. 3, our RGB-D pretraining brings 2.3% improvement for DFormer-B compared to the RGB pretraining in terms mloU on NYU D2 We argue that this because our RGB- D pretraining avoids the mismatch encoding of the 3D geometry features of depth maps caused by the use of pretrained RGB backbones and enhances the interaction efficiency between the two modalities. Tab. 12 and Fig. 7 in the appendix also demonstrate this. These experimental results indicate that the RGB-D representation capacity learned during the RGB-D pretraining is crucial for segmentation accuracy.
Input features of the decoder. Benefiting from the powerful RGB-D pretraining, the features of the RGB branch can efficiently fuse the information of two modalities. Thus, our decoder only uses the RGB features Xr9b, which contains expressive clues, instead of using both r and X As shown in Tab. 4, using only Xrgb can save computational cost without performance drop, while other methods usually need to use both Xr and X This difference also demonstrates that our proposed RGB-D pretraining pipeline and block are more suitable for RGB-D segmentation.
Components in our RGB-D block. Our RGB-D block is composed of a base module, GAA mod- ule, and LEA module. We take out these three components from DFormer respectively, where the results are shown in Tab. 5. It is clear that all of them are essential for our DFormer. Moreover, we visualize the features around the RGB-D block in Fig. It can be seen the output features can capture more comprehensive details. As shown in Tab. 7, in the GAA module, we find that only fusing the depth features into Q is adequate and further fusing depth features to and V brings negligible improvement but extra computational burden. Moreover, DForer initially rises as the fixed pooling size of the GAA increases, and achieves the best when the fixed pooling size is set to 7x7 in Tab. 6. We also use other fusion manners replace that
find that the performance of

---

Published as a conference paper at ICLR 2024
Table Comparisons under the RGB-only pre-
Table 10: Comparisons under the RGB-D pre-
training. 'NYU" and 'SUN' means the perfor-
training. 'NYU" and 'SUN' means the perfor- mance on NYU Depth and SUNRGBD.
mance on the NYU D2 and SUNRGBD.
Model
Params FLOPs
NYU
SUN
Model
Params FLOPs
NYU
SUN
CMX (MiT-B2)
66.6M
67.6G
54.4
49.7
CMX (MiT-B2)
66.6M
67.66
55,8
51.1
DFormer-B
29.5M
41.9G
53.3
49.5
DFormer-B
29.5M
41.9G
55.6
51.2
DFormer-L
39.0M
65.7G
55.4
DFormer-L 39,0M
65.7G
57.2
52.5
in LEA, i.e., concatenation, addition. As shown in Tab. 8, using the depth features that processed by a large kernel depth-wise convolution as attention weights to reweigh the RGB features via simple Hadamard product achieves the best performance.
Channel ratio between RGB and depth. RGB images contain information pertain- ing to object color, texture, shape, and its surroundings. In contrast, depth im-
mloU
Hops
Purumetur
Pasmzters(M) Hlopss
30.0
inooU
53.0
ages typically convey the distance infor-
52.0
mation from each pixel to the camera. Here, we investigate the channel ratio that used to encode the depth information. In Fig. 6, we present the performance of DForeer-S with different channel ratios,
20.0
51.0
i.e, Cd/Crbb, We can see that when the
10.0
1/8
1/
1/2
3/4
channel ratio excceds '1/2', the improve- ment is trivial while the computational burden significantly increased. There- fore, we set the ratio to 1/2 by default.
Figure 6: Porformance on different channel ratios Ct aC based on DFormer-S. Crabb is fixed and we adjust Crd to get different ratios.
Applying the RGB-D pretraining manner to CMX. To verify the effect of the RGB-D pretraining on other methods and make the comparison fairer, we pretrain CMX (MiT-B2) on the RGB-D data of ImageNet and obtains about 1.4% mloU improvement, as shown in Tab. and Tab. 10. Under the RGB-D pretraining, DFormer-L still outperforms CMX (MiT-B2) by large margin, which should be attributed to that the pretrained fusion weight within DFormer can achieve better and efficient fusion between RGB-D data. Besides, provide the RGB pretrained DFormers to provide more insights Tab. 9. similar phenomenon appears under the RGB-only pretraining.
Discussions on the generalization ability to other modalities. Through RGB-D pretrain- ing, DFormer is endowed with the capac- ity to interact the RGB and depth features dur- ing pretraining. To verify whether the interac- tion method still works when replacing depth with another modality, we apply our DFormer to some benchmarks with other modalities, ie., RGB-T on MFNet (Ha et al., 2017) and RGB-L on KITTI-360 (Liao et al., 2021). As shown in Tab. 11 (Comparisons to more meth- ods are shown in Tab. 14 and Tab. 15), our pre- training framework still improves the perfor- mance. However, the improvement is limited compared to that on RGB-D scenes. To address this issue, foreseeable solution is conduct pretraining of DFormer on other modalities. We will view this as our future work.
Table 11: Results on the RGB-T semantic seg- mentation benchmark MFNet (Ha et al., 2017) and RGB-L semantic segmentation benchmark KITTI-360 (Liao et al., 2021). '(RGB) and '(RGDD)' mean RGB-only and RGB-D pretrain- ing, respectively. .*1 represents our implementa- tions.
Model
Params
FLOPs
MFNet
KITTI
CMX-B2 B2
66.6M
67.60
58.2
64.3
CMX-B4
139.9M
134.3G
59.7
65.5*
CM2
65.1M
65.5G
58.4*
65.3
CMNeXt-B4
135.6M
132.6G
59.9
65.6*
Ours-L (RGB)
39.0M
65.7G
59.5
65.2
Ours-L (RGBD)
39.0M
65.7G
60.3
66.1
RELATED WORK
RGB-D scene parsing. In recent years, with the rise of deep learning technologies, e.g., CNNs (He et al., 2016), and Transformers (Vaswani et al., 2017; 7; Li et al., 2023a), significant progress has been made in scene parsing (Xie al., 2021; Yin et al., 2022; Chen al., 2023; Zhang al., 2023d),
8

---

Published as a conference paper at ICLR 2024
one of the core pursuits of computer vision. However, most methods still struggle to cope with some challenging scenes in the real world (Li et al., 2023b; Sun et al., 2020), as they only focus on RGB images that provide them with distinet colors and textures but no 3D geometric information. To overcome these challenges, researchers combine images with depth maps for a comprehensive understanding of scenes.
Semantic segmentation and salient object detection are two active areas in RGB-D scene parsing. Particularly, the former aims to produce per-pixel category prediction across given scene, and the latter one attempts to capture the most uutention-grabbenn and alignment between RGB-D modalities, the prevalent methods investigate lot of efforts in building fusion modules to bridge the RGB and depth features extracted by two parallel pretrained backbones. For example, methods like CMX (Zhang et al., 2023a), TokenFusion (Wang et al., 2022), and HiDANet (Wu et al., 2023) dynamically fuse the RGB-D representations from RGB and depth encoders and aggregate them the decoder. The evolution of fusion manners has dramatically pushed the performance boundary in these applications of RGB-D scene parsing. Nevertheless, the three common issues, as discussed in Sec. 1, are still left unresolved.
objects. To achieve the interaction
Another line of works focuses on the design of operators (Wang & Neumann, 2018; Wu et al., 2020; Cao et al., 2021; Chen et al., 2021a) to extract complementary information from RGB-D modal- ities. For instance, methods like ShapeConv (Cao et al., 2021), SGNet (Chen al., 2021a), and Z-ACN (Wu et al., 2020) propose depth-aware convolutions, which enable efficient RGB features and 3D spatial information integration largely enhance the capability of perceiving geometry. Al- though these methods are efficient, the improvements brought by them are usually limited due to the insufficient extraction and utilization of the 3D geometry information involved in the depth modal.
Multi-modal learning. The great success of the pretrain-and-finett processing and computer vision has been expanded to the multi-moda domain, and the learned trans- ferable representations have exhibited remarkable performance on wide variety of downstream tasks. Existing multi-modal learning methods cover a large number of modalities, e.g., image and text (Castrejon et al., 2016; Chen et al., 2020b; Radford et al., 2021; Zhang et al., 2021b; Wu et al., 2022), text and video (Akbari et al., 2021), text and 3D mesh (Zhang et al., 2023c), image, depth, and video (Girdhar et al., 2022). These methods can be categorized into two groups, i.e., multi- and joint-encoder ones. Specifically, the multi-encoder methods exploit multiple encoders to indepen- dently project the inputs in different modalities into common space and minimize the distance or perform representation fusion between them. For example, methods like CLIP (Radford et al., 2021) and VATT (Akbari al., 2021) employ several individual encoders embed the representations in different modalities and align them via a contrastive learning strategy. In contrast, the joint-encoder methods simultaneously input the different modalities and use multi-modal encoder based on atten- tion mechanisms to model joint representations. For instance, MultiMAE (Bachmann et al., 2022) adopts unified transformer to encode the tokens with a fixed dimension that are linearly projected from a small subset of randomly sampled multi-modal patches and multiple task-specific decoders to reconstruct their corresponding masked patches by attention mechanisms separately.
paradigm in natural language
In this paper, we propose DFormer, a novel framework that achieves RGB-D representation learning in a pretraining manner. To the best of our knowledge, this is the first attempt to encourage the semantic cues from RGB and depth modalities to align together by the explicit supervision signals of classification, yielding transferable representations for RGB-D downstream tasks.
CONCLUSIONS
In this paper, we propose novel RGB-D pretraining framework to learn transferable representa- tions for RGB-D downstream tasks. The core of our model is a tailored RGB-D block. Thanks to the RGB-D block, our method is able to achieve better interactions between the RGB and depth modalities during pretraining. This avoids the use of another branch for encoding depth informa- tion. Our experiments show that DFormer can achieve new state-of-the-art performance in RGB-D downstream tasks, e.g., semantic segmentation and salient object detection, with far less computa- tional cost compared to existing methods. We also show that our pretraining framework works also well for other modalities, like RGB-T and RGB-L. We hope this work could bring the multi-modal segmentation community insights for advanced model design.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research was supported by National Key Research and Development Program China (No. 2021 MEI3IB33100 (00000, NSFC (NO. 62225604, No. 62276145, and No. 62376283), the Fundamental Research Funds for the Central Universities (Nankai University, 070-63223049, CAST through Young Elite Scientist Sponsorship Program (No. YESS20210377). Computations were supported by the Supercomputing Center of Nankai University (NKSC).
REFERENCES
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NearlPS, 2021. 9
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-modal multi- task masked autoencoders. ECCV, 2022. 5,9
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adap- tive bins. In CVPR, 2021. 2, 4, 16
Tim Broedermann, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. HRFuser: A multi- resolution sensor fusion architecture for 2D object detection. arXiv preprint arXiv:2206. 15157, 2022. 18
Jinming Cao,, Hanchao Leng, Dani Lischinski, Danny Cohen-Or, Changhe Tu, and Yangyan Li. ShapeConv: Shape-aware convolutional layer for indoor RGB-D semantic segmentation. In ICCV, 2021. 5,
Lluis Castrejon, Yusuf Aytar, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Learning aligned cross-modal representations from weakly aligned data. In CVPR, 2016. 9
Lin-Zhuo Chen, Zheng Lin, Ziqin Wang. Yong-Liang Yang, and Ming-Ming Cheng. Spatial in- formation guided convolution for real-time RGBD semantic segmentation. TIP. 00:2313-2324, 2021a. 5,9
Qian Chen, Zhenxi Zhang, Yanye Lu, Keren Fu, and Qijun Zhao. 3-d convolutional neural net- works for rgb-d salient object detection and beyond. IEEE Transactions Neural Networks and Learning Systems, 2022.
Xiaokang Chen et al. Bi-directional cross-modality feature propagation with separation-an aggregation gate for RGB-D semantic segmentation. In ECCV, 2020a. 5, 18
Xinghao Chen, Chang Xu, Minjing Dong, Chunjing Xu, and Yunhe Wang. An empirical study of adder neural networks for object detection. NeurIPS, 2021b. 16
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020b.
Yuming Chen, Xinbin Yuan, Ruiqi Wu, Jiabao Wang, Qibin Hou, and Ming-Ming Cheng. Yolo- ms: Rethinking multi-scale representation learning for real-time object detection. arXiv preprint 0rXiv:2308.055880 2023. 8
Yupeng Cheng, Huazhu Fu, Xingxing Wei, Jiangjian Xiao, and Xiaochun Cao. Depth enhanced saliency detection method. In ICIMCS, 2014. 20
Fuqin Deng FEANet: Feature-enhanced attention network for RGB-thermal real-time semantic segmentation. In IROS, 2021. 18
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In ICCV, 2021. 16
Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Structure-meauure: measure: A new way to evaluate foreground maps. In IEEF ICCV, 2017. 6, 20
10

---

Published as a conference paper at ICLR 2024
Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng. and Ali Borji. Enhanced- alignment measure for binary foreground map evaluation. ICCAI, 2018. 6, 20
Deng-Ping Fan, Zheng Lin, Zhao Zhang, Menglong Zhu, and Ming-Ming Cheng. Rethinking rgb-d salient object detection: Models, data sets, and large-scale benchmarks. TNNLS, 32(5):2075- 2089, 2020. 6, 20
Oriel Frigo, Lucien Martin-Gaffe, and Catherine Wacongne. Doodlenet: Double deeplab enhanced feature fusion for thermal-color semantic segmentation. In CVPR, 2022. 18
Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? NeurlPS, 2021. 5, 20
Anjith George and Sebastien Marcel. Cross modal focal loss for rgbd face anti-spoofing. In CVPR, 2021. 21
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In CVPR, 2022. 5, 9.9
Meng- Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. NeurIPS, 2022a. 5, 19, 20
Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual atten- tion network. arXiv preprint arXhv:2202.09741) 2022b. 4
Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes, In IROS, 2017. 8, 18
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Qibin Hou, Cheng-Zo Lu, Ming-Ming Cheng, and Jiashi Feng. Conv2former: A simple transformer- style convnet for visual recognition. arXiv preprint arXiv:2211.11943, 2022.
Xinxin Hu, Kailun Yang, Lei Fei, and Kaiwei Wang. ACNet: Attention based network to exploit complementary features for RGBD semantic segmentation. In ICIP, 2019.
Keli Huang, Botian Shi, Xiang Li, Xin Li, Siyuan Huang, and Yikang Li. Multi-modal sensor fusion for auto driving perception: A survey, arXiv preprint arXiv:2202.02277 022202002773,. 2022. 1
Wei Ji, Jingjing Li, Shuang Yu, Miao Zhang, Yongri Piao, Shunyu Yao, Qi Bi, Kai Ma. Yefeng Zheng, Huchuan Lu, al. Calibrated rgb-d salient object detection. In CVPR, 2021.
Ran Ju, Ling Ge, Wenjing Geng, Tongwei Ren, and Gangshan Wu. Depth saliency based on anisotropic center-surround difference. In ICIP, 2014. 6, 20
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015. 5, 20
Minhyeok Lee, Chaewon Park, Suhwan Cho, and Sangyoun Lee. Spsn: Superpixel prototype sam- pling network for rgh-d salient object detection. In ECCV, 2022. 6
Gongyang Li, Yike Wang, Zhi Liu, Xinpeng Zhang, and Dan Zeng. Rgb-t semantic segmentation with location, activation, and sharpening. IEEE TCSVT, 33(3):1223-1235 2022. 18
Zhong-Yu Yu Li, Shanghua Gao, and Ming-Ming Cheng. Sere: Exploring feature self-relation for self-supervised transformer. IEEE TPAMI, 2023a. 8
Zhong- Yu Li, Bo-Wen Yin, Shanghua Gao, Yongxiang Liu, Li Liu, and Ming-Ming Cheng. Enhancing representations through heterogeneous self-supervised learning. arXiv preprint arXiv:2310.05 2023b.
11

---

Published as a conference paper at ICLR 2024
Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D. arXiv preprint arXiv:2109. 2021. 8, 18
Haolin Liu, Anran Lin, Xiaoguang Han, Lei Yang, Yizhou Yu, and Shuguang Cui. Refer-it-in-rgbd: A bottom-up approach for 3d visual grounding in rgbd images. In CVPR, 2021a. 21
Nian Liu, Ni Zhang. Kaiyuan Wan, Ling Shao, and Junwei Han. Visual saliency transformer. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 4722-4732, 2021b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. In CVPR, 2022. 4
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 4
Nicolas Marchal, Charlotte Moraldo, Hermann Blum, Roland Siegwart, Cesar Cadena, and Abel Gawel. Learning densities in feature space for reliable segmentation of indoor scenes, RA-L, (2):1032-1038, 2020.
Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal. How to evaluate foreground maps? In IEEE CVPR, 2014. 6, 20
Yuzhen Niu, Yujie Geng, Xueqing Li, and Feng Liu. Leveraging stereopsis for saliency analysis. In CVPR, 2012. 6,
Houwen Peng, Bing Li, Weihua Xiong, Weiming Hu, and Rongrong Ji. Rgbd salient object detec- tion: A benchmark and algorithms. In ECCV, 2014. 6, 20
Federico Perazzi, Philipp Krähenbühl, Yael Pritch, and Alexander Hornung, Saliency filters: Con- trast based filtering for salient region detection. In JEEE CVPR, 2012. 6, 20
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to- end autonomous driving. In CVPR, 2021. 18
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021. 9
Olga Russakovsky al. ImageNet large scale visual recognition challenge. IJCV, 115(3):11-252, 2015. 4, 16
Daniel Seichter, Mona Köhler, Benjamin Lewandowski, Tim Wengefeld, and Horst-Michael Gross. Efficient rgb-d semantic segmentation for indoor scene analysis. In ICRA, 2021. 5
Daniel Seichter, Söhnke Benedikt Fischedick, Mona Köhler, and Mirst-Michael GroB, Efficient multi-task rgb-d scene analysis for indoor cavironments. In IJCNN, 2022.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from RGBD images. ECCV, 2012. 2, 5, 7, 19, 20, 22
Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. SUN RGB-D: A RGB-D scene under- standing benchmark suite. In CVPR, 2015. 5, 6, 19, 20
Peng Sun, Wenhu Zhang, Huanyu Wang, Songyuan Li, and Xi Li. Deep rgb-d saliency detection with depth-sensitive attention and automatic multi-modal fusion. In CVPR, 2021a. 6
Xiaoshuai Sun, Xuying Zhang, Liujuan Cao, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Ex- ploring language prior for mode-sensitive visual attention modeling. In Proceedings of the 28th
ACM International Conference on Multimedia, pp. 4199-4207, 2020. 9
Yuxiang Sun, Weixun Zuo, Peng Yun, Hengli Wang, and Ming Liu. FuseSeg: Semantic segmen- tation of urban scenes based on RGB and thermal data fusion. T-ASE, 1000)1000--011, 2021b.
18
12

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017. 3, 4,
Hengyi Wang, Jingwen Wang, and Lourdes Agapito. Co-slam: Joint coordinate and sparse paramet- ric encodings for neural real-time slam. In CVPR, 2023. 1
Weiyue Wang and Ulrich Neumann. Depth-aware enn for rgb-d segmentation. ECCV, 2018.
Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong. and Junzhou Huang. Deep multimodal fusion by channel exchanging. NeurIPS, 2020. 5
Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang. Multimodal token fusion for vision transformers. In CVPR, 2022. 2, 5, 9, 15, 18, 20
Hongfa Wen, Chenggang Yan, Xiaofei Zhou, Runmin Cong, Yaoqi Sun, Bolun Zheng, Jiyong Zhang, Yongjun Bao, and Guiguang Ding. Dynamic selective network for rgb-d salient object detection. IEEE TIP, 30:9179-9192, 2021. 6
Mingrui Wu, Xuying Zhang, Xiaoshuai Sun, Yiyi Zhou, Chao Chen, Jiaxin Gu, Xing Sun, and Rongrong Ji. Difnet: Boosting visual information flow for image captioning. In CVPR, 2022. 9
Zongwei Wu, Guillaume Allibert, Christophe Stolz, and Cédric Demonceaux. Depth-adapted cnn for rgb-d cameras. In ACCV, 2020. 9
Zongwei Wu, Guillaume Allibert, Fabrice Meriaudeau, Chao Ma, and Cédric Demonceaux. Hidanet: Rgb-d salient object detection via hierarchical depth awareness, IEEE TIP, 32:2160-2173, 2023. 6, 9, 23
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Seg- Former: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021. 5, 8, 19
Song Yan, Jinyu Yang, Jani Käpylä, Feng Zheng, Aleš Leonardis, and Ooii-Kristian Kämäräinen. Depthtrack: Unveiling the power of rgbd tracking. In ICCV, 2021. 21
Yang Yang, Qi Qin, Yongjiang Luo, Yi Liu, Qiang Zhang, and Jungong Han. Bi-directional pro- gressive guidance network for rgb-d salient object detection. IEEE Transactions on Circuits and Systems for Video Technology, 32(5):5346-5360 2022. 6
Bowen Yin, Xuying Zhang, Qibin Hou, Yu-Y Sun, Deng-Ping Fan, and Lue Van Gool. Camoformer: Masked separable attention for camouflaged object detection. arXiv preprint arXiv:2212.06570 2022. 8
Tao Yu, Zerong Zheng. Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In CVPR, 2021. 21
Yingjie Zhai, Deng-Ping Fan, Jufeng Yang, Ali Borji, Ling Shao, Junwei Han, and Liang Wang. Bifurcated backbone strategy for rgb-d salient object detection. IEEE TIP, 30:8727-8742, 2021.
Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen. Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers. T-ITS, 2023a. 5, 9, 15, 18, 20
Jiaming Zhang, Ruiping Liu, Hao Shi, Kailun Yang, Simon Reiß, Kunyu Peng, Haodong Fu, Kaiwei Wang, and Rainer Stiefelhagen. Delivering arbitrary-moaal semantic segmentation. In CVPR, 2023b. 2, 5, 6, 18, 20, 22
Jing Zhang, Deng-Ping Fan, Yuchao Dai, Xin Yu, Yiran Zhong, Nick Barnes, and Ling Shao. RGB- D saliency detection via cascaded mutual information minimization. In ICCV, 2021a. 6
Miao Zhang, Shunyu Yao. Beiqi Hu, Yongri Piao, and Wei Ji. dfdnnt: Criss-cross dynamic filter network for rgb-d salient object detection. IEEE TMM, 2022. 6
13

---

Published as a conference paper at ICLR 2024
Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Rstnet: Captioning with adaptive attention on visual and non-visual words. In CVPR, 2021b. 9
Xuying Zhang, Bo-Wen Yin, Yuming Chen, Zheng Lin, Yunheng Li, Qibin Hou, and Ming-Ming Cheng. Temo: Towards text-driven 3d stylization for multi-object meshes. arXiv preprint arXiv:2312.04248, 2023c.
Xuying Zhang, Bowen Yin, Zheng Lin, Qibin Hou, Deng-Ping Fan, and Ming-Ming Cheng. Refer- ring camouflaged object detection. arXiv preprint irXiv:2306.07532, 2023d.
Zhao Zhang, Zheng Lin, Jun Xu, Wen-Da Jin, Shao-Ping Lu, and Deng-Ping Fan. Bilateral attention network for rgb-d salient object detection. IEEE 77P, 30:1949-1966 2021c. 6
Yifan Zhao, Jiawei Zhao, Jia Li, and Xiaowu Chen. Rgb-d salient object detection with ubiquitous target awareness. IEEE TIP, 30:7717-7731, 2021.
Heng Zhou, Chunna Tian, Zhenxi Zhang, Qizheng Huo, Yongqiang Xie, and Zhongbo Li. Multi- spectral fusion transformer network for rgb-thermal urban scene semantic segmentation. IEEE GRSI,, 19:1-5, 2022a. 18
Jiayuan Zhou, Lijun Wang, Huchuan Lu, Kaining Huang, Xinchu Shi, and Bocong Liu. Mvsalnet: Multi-view augmentation for rgb-d salient object detection. In ECCV, 2022b. 6
Tao Zhou, Huazhu Fu, Geng Chen, Yi Zhou, Deng-Ping Fan, and Ling Shao. Specificity-preserving rgb-d saliency detection. In ICCV, 2021a. 6, 20, 23
Wujie Zhou, Jinfu Liu, Jingsheng Lei, Lu Yu, and Jenq-Neng Hwang. GMNet: Graded-featus multilabel-carriing network for RGB-thermal urban scene semantic segmentation. TIP, 30:7790- 7802, 2021b. 18
Wujie Zhou, Enquan Yang. Jingsheng Lei, Jian Wan, and Lu Yu. Pgdenet: Progressive guided fusion and depth enhancement network for rgb-d indoor scene parsing. IEEE TMM, 2022c. 5
Wujie Zhou, Enquan Yang, Jingsheng Lei, and Lu Yu. Frnet: Feature reconstruction network for rgb-d indoor scene parsing. JSTSP, 16(4):677-687, 2022d.
Zhuangwei Zhuang, Rong Li, Kui Jia, Qicheng Wang, Yuanqing Li, and Mingkui Tan. Perception- aware multi-sensor fusion for 3d lidar semantic segmentation. In ICCV, 2021. 18
14

---

Published as a conference paper at ICLR 2024
APPENDIX
In Sec. A, we first provide further analysis of our DFormer: 1) The efficiency in encoding the depth maps; 2) The effect of depth maps with different quality during pretraining. Then we present more detailed descriptions of DFormer in Sec. B and the experimental settings in Sec. C. Finally, we provide more visualization results of DFormer in Sec. and future exploring directions Sec. E.
A MORE ANALYSIS ON DFORMER
RGB Backbone
Depth
Backbone
Before finetuning
Before finetuning
After finetuning
After finetuning
(a) RGB backbonc
(b) Depth backbone
Figure 7: Encoding depth maps with backbones pretrained on different types of training data. (a) Pretraining with RGB data. (b) Pretraining with depth maps. During finetuning, only take the depth maps input to see which backbone works better. We visualize some features from the two backbones shown in the bottom part. Obviously, the backbone pretrained on depth maps can generate more expressive feature maps.
Performance of the RGB pretrained and depth pretrained backbone for segmentation. The backbones adopt same pretraining settings and architecture hut are pretrained ImageNet images and their depth maps, respectively.
Table 12:
Backbone
#Params
FLOPs
mloU(%%
RGB
11.2M
13.7G
27.6
Depth
11.2M
13.7G
42.8
Why involving depth information in pretraining? The existing state-of-the-art RGB-D meth- ods (Zhang et al., 2023a; Wang et al., 2022) tend
models pretrained RGB images encode 3D geometry information for depth maps. We argue that the huge representation distribution shift caused by using RGB backbone to encode the depth maps may influence the extraction of the 3D ge- ometry. To demonstrate this, we respectively use RGB and depth data to pretrain the RGB and depth backbone and then take only the depth maps as input for segmentation, as shown in Fig. From Tab. 12, we can see under the same network architecture, the model pretrained on depth images performs much better than the one pretrained on RGB images, i.e., yielding an improvement mloU. To delve into the reasons, visualize the feature maps shown in the bottom part of Fig. 7. Before finetuning, the backbone pretrained on RGB data is not able to extract expressive features from the depth maps. After finetuning, the model using RGB backbones still struggles to extract diverse features from the depth maps. On the contrary, the features from the backbone pretrained on depth data better. These experiments indicate that there exists significant difference between
15%
15

---

Published as a conference paper at ICLR 2024
RGB and depth maps and it is difficult to process the depth data with RGB pretrained weights. This also motivates us to involve depth data during ImageNet pretraining.
Image
BENERIC
MLET TISSE
ASSHI
Adabins
Omnidata
Figure 8: Comparisons of estimated depth maps that generated by Adabins (Bhat et al., 2021) and more advanced Omnidata (Eftekhar et al., 2021). We visualize the depth maps in color for better comparisons.
Impact on the quality of depth maps. In the main paper, we use depth estimation method Adabins (Bhat al., 2021) to predict depth maps for the ImageNet- IK (Russakovsky et al., 2015) dataset. Both the ImageNet images and the generated depth maps are used to pretrain our DFormer. Here, we explore the effect of depth maps in different quality on the performance of DFormer. To this end, we also choose more advanced depth estimation method, ie., Omnidata (Eftekhar et al., 2021), to generate the depth maps. Fig. 8, we visualize the generated depth maps that generated by these two methods. We use the Ommidata-predicter depth maps replace the Adabins-predicted predicted ones during RGB-D pretraining. Then, we use this to pretrain the DFormer- Base and it achieves nearly the
Adabins-
performance, i.e., 55.6 mloU on NYUDepth v2. This indicates DFormer is robust to the quality of depth maps.
RGB pretrained Drorme
RGB pretrained Dromer
BN namning man
Finetuned Pretrained
Rinetuned Protrained
20
40
60
80
100
120
100
Pinetuned Pretrained
Finctuned Pretreined
mimingg
1000
80
120
20
40
60
100
120
Figure 9: The statistics of distribution shift on the finetuning of DFormer-S that uses different pretraining manners. The BN layer the first block of Stage 2 is chosen for this visualization.
Observations towards the distribution shift. Interacting the RGB and depth features within the pre-trret encoders would bring in drastic changing of the feature distribution, which makes the previous statistics of batch normalization incompatible with the input features. Following (Chen et al., 2021b), we visualize the statistics of the BN layers during the finetuning process to reflect the distribution shift. Specifically, Fig. 9 to observe the statistics of the fused features. For the RGB-D pretrained DFormer, the running mean and variance of the BN layer only have slight changes after finetuning, illustrating the learned RGB-D representations are transferable for the RGB-D segmentation tasks. In contract, for the RGB pretrained one, the statisties of the BN layer changed sharply after finetuning, which indicates
visualize the BN layer for a random layer from DFormer in
16

---

Published as a conference paper at ICLR 2024
Images
Features in RGBD pretrained model
Features in RGB pretrained model
Visualization of the features within the finetuned models that load RGB -rrerrrtadneed and RGBD- Figure 10:
pretrained weights. The features are randomly picked from the first stage output in DFormer-S.
the encoding is mismatched. The situation forces RGB pretrained weights to adapt the input features fused by the two modalities. We also visualize some features of DFormer that uses the RGB and RGB-D pretraining, as shown in Fig. 10.
B DETAILS OF DFORMER
Structure, In the main paper, we only present the structure map of the interaction modules due to the limited space. The detailed structure of our RGB-D block is presented in Fig. 11. The GAA, LEA and base modules comprise the block, and each of them is essential and contributes to the per- formance improvement. GAA and LEA aims to conduct interactions between different modalities globally and locally, while the base module is only responsible for encoding RGB features. As the first stage focuses encoding low-level features, do not use GAA in the first stage for the sake of efficiency. Moreover, the MLP layers for the RGB and depth features are independent.
Hadamard Product
XXX SAAX X CT
x
rgb
out
X
out
Ca
Concat
Feature Concatenation
X
DX W
Pool
Spatial Pooling
Proj.
Proj.
Concat
Global Aware
Conv
DWConv
Attention
hxwXC Cd
CTBM
We
Concat Pool
Wx, Wy
W B
Wa
Wr
W
GAA
LEA
h
cd
Base
x
x
x
x
x
Tgg
Figure 11: Detailed structure of our RGB-D block DFormer.
17

---

Published as a conference paper at ICLR 2024
Inference time analysis. Real-time in- ference an RGB-D model plays a key role in wide spectrum of downstream applications, as stated by (Chen et al., 2020a). To this end, we conduct exper- iments to explore the real-time potential of our DFormer and other methods. To ensure fairness, all comparisons are per- formed on the same device, i.e,, sin- gle 3090 RTX GPU, and the same im- age resolution, iee, 480 x 640. As illus- trated in Fig. 12, our DFormer-L achieves 57.2% mIoU with 35.7 ms latency, while the latency of the current nate-of-the- art CMNext is 78.2 ms. Remarkably, our DFormerSS process image within 20ms and achieve about 50 frames per sec- ond (FPS) with competitive performance on NYU Depthv2, i.e., 53.6% mloU.
Ows
57
CMNext
MulaMAE
56
-B
CMX
55
TokenFusion
mloo
54
53
Omnivore
52
Model
mfol]
Parum
Floge:
Labracy
MaiiMAE
56.0
95.2M
267.9G
19.9ms
51
167.8G
CMNext-B4
56.9
119.6M
131.96
78.2ms
50
Ours-l
57.3
39.0M
65.76
35.7ms
20
30
40
50
60
70
80
90
100
Latency (ms)
Figure 12: Performance vs.
latency when processing
x 640 images.
Table 13: Ablation towards the base modules within the building block of DFormer-S.
DWConv Setting
Attention Operation
Param
Flops
NYUDepthV2
DWConv x
Hadamard Product
18.7M
25.6G
51.9
DWConv 5 x
Hadamard Product
18.7M
23.9G
51.6
DWConv 9 x
Hadamard Product
18.7M
27.1G
51.9
DWConv x
Addition 18.7M
25.0G
51.3
DWConv x
Concatanation 19.3M
26.9G
51.7
More detailed ablations on the block. In the main paper, we have provided the ablation exper- iments about the components of our RGB-D block, the pooling size of our GAA, as well as the fusion manners in GAA and LEA. Here we provide more results for the modules that encode RGB features, as shown in Tab. 13. All the models are pretrained for 100 epochs using DFormer-S. Note that the results in the table only ablates the structure within the module (gray part in Fig. 11) that only processes the RGB features.
Table 14: Results on MFNet (RGB-T) (Ha al., 2017).
Table 15: Results on KITTI-360 (RGB-L) (Liao
et al.).
Method
Backbone
mloU
Method
Backbone
mloU
FuseSeg (Sun al.)
DenseNet-161
54.5
LASNet (Li et al.)
ResNet-152
54.9
HRFuser (Broedermann al.)
HRFormer-T
48.7
FEANet (Deng et al.)
ResNet-152
55.3
PMF (Zhuang et al.)
SalsaNext
54.5
MFTNet (Zhou et al.)
ResNet-152
57.3
GMNet (Zhou et al.)
ResNet-50
57.3
TokenFusion (Wang et al.)
MiT-B2
54.6
LeNet (Frigo et al.)
ResNet-101
57.3
TransFuser (Prakash al.)
RegNetY
56.6
CMX (Zhang al.)
MiT-B2
58.2
CMX (Zhang et al.)
MiT-B2
64.3
CMX (Zhang et al.)
MiT-B4
59.7
CMNeXt (Zhang al.)
MiT-B2
65.3
CMNeXt (Zhang al.)
MiT-B4
59.9
(RGB) DFormer
Ours-L
65.2
(RGB) DFormer
Ours-L
59.5
(RGB D) DFormer
Ours-L
60.3
(RGB-D) DFormer
Ours-L
66.1
B. COMPARISON ON THE MENET AND KITTI-360
For comprehensive comparison, compare DFormer and more other state-of-the- art methods on MFNet (Ha et al., 2017) and KITTI-360 (Liao et al., 2021) in Tab. 15 and Tab. 14, supplement to the main paper.
18

---

Published as a conference paper at ICLR 2024
Table 16: Detailed configurations of the proposed DFormer. "C" ""Crg" 'Cd'', which respectively represent the channel number of the RGB part and the depth part in different stages. 'Ny' is the number of building blocks i-th stage. "Expansion' is the expand ratio for the number of channels in MLPs. 'Decoder dimension' denotes the channel dimension in the decoder.
Stage
Output size
Expansion
DForme--T
DFormer-S
DFormer-B
DFormer-L
Stem
X
C= (16,8)
(32, 16)
(32, 16)
(18, 24)
1
X
(32, 16), Nı 3
(64,32), 32), 2
(64, (2,,32)
(96, 48),
2
X
C 32), N2 3
(128, 64), 2
(128, 64),
(192, 96), 3
3
18
X
16
C (128, 64), N2 5
(256, 128),
(256, 128), 12
(288, 144), 12
4
C' (256, 128). Nu = (512, 256).
(512, 256). 2
(576, 288).
32
32
Decoder dimension
512
512
512
512
Parameters (M)
6.0
18.7
29.5
39.0
C EXPERIMENTAL DETAILS
C.I IMAGENET PRETRAINING
We provide DFormers' ImageNet-IK pretraining settings in Tab. 17. All DFormer variants the same settings, except the stochastic depth rate. Besides, the data augmentation strategies related to color, e.g., auto contrast, are only used for RGB images, while other common strategies are simultaneously performed on RGB images and depth maps, e.g., random rotation.
Table 17: DFormer ImageNet-1K -K pretraining settings. All the pretraining experiments are con- ducted on 8 NVIDIA 3090 GPUs
pretraining config
DFomer-T
DForer-S
DFormer-B
DFormer-L
input size
2242
224
224
224
weight
trunc. normal (0.2)
trunc. normal (0.2)
trune. normal (0.2)
trunc. normal (0.2)
optimizer
AdamW
AdamW
Adam
AdamW
base leariing rate
le-3
1e-3
le-3
1e-3
weight decay
0.05
0.05
0.05
0.05
optimizer momentum
81, 9.=0.99 0.999
81, 82=0.,9 0.999
82=09, 0.999
= 0.999
batch size
1024
1024
1024
1024
training epochs
300
300
300
300
learning rate schedule
cosine decay
cosine decay
cosine decay
cosine decay
warmup epochs warmup schedule
5
5
5
linear
linear
linear
linear
layer-wise Ir decay
None
None
None
None
randaugment
(9, 0.5)
(9, 0.5)
(9, 0.5)
(9, 0.5)
mixup
0.8
0.8
0.8
0.8
cutmix
1.0
1.0
1.0
random erasing
0.25
0.25
0.25
0.25
label smoothing
0.1
0.1
stochastic depth
0.1
0.1
0.15
0.2
head init scale
None
None
None
Nonc
gradient clip exp. mov. avg. (EMA)
None
None
None
None
None
None
None
None
C.2 RGB-D SEMANTIC SEGMENTATION FINETUNING
The finetuning settings for NYUDepth v2 and SUNRGBD datasets are listed in Tab. 18. The batch sizes, input sizes, base learning rate, epochs and stochastic depth are different for the two datasets.
Datasets. Following the common experiment settings of RGB-D semantic segmentation meth- ods (Xie et al., 2021; Guo et al., 2022a), we finetune and evaluate DFormer on two widely used datasets, Lee. NYUDepthy2 (Silberman al., 2012) and SUN-RGBD (Song al., 2015). To be spe-
19

---

Published as a conference paper at ICLR 2024
Table 18: DFormer finetuning settings on NYUDepttV//UNRGGBD Multiple stochastic depth
rates, input sizes and batch sizes are for NYUDepthv2 and SUNRGBD datasets respectively. All the finetuning experiments for RGB-D semantic segmenations are conducted on NVIDIA 3090 GPUs.
pretraining config
D
DForme--S
DForer--
DFormer-L.
input size
480 x 640 480
480
640 480
480 x 640/ 1802
480 x 640 /80
optimizer
AdamW
AdamW
AdamW
AdamW
base learning rate
6e-5/8e-5
6e-5/8e-5
6e-5/8e-5
6e-5/8e-5
weight decay
0.01
0.01
0.01
0.01
batch size
8/16
8/16
8/16
8/16
epochs
500/300
500/300
500/300
500/300
optimizer momentum
31, 0.9, 0.999
81. B2 -0.9, 0.999
3 (2200.9 0.999
B1, 30.9, 0.999
training epochs
300
300
300
300
learning rate schedule
lincar decay
linear decay
linear decay
lincar decay
warmup epochs
10
10
10
10
warmup schedule
linear
linear
linear None
linear
layer-wise wise decay
None
None
None
aux head stochastic depth
None
None
None
None
1/1/0.1
0.1/0.1
0.1/0.1
0.15/0.2
cific, NYUDepthv2 (Silberman et al., 2012) contains 1, 1,449 RGB-D samples covering 40 categories, where the resolution of all RGB images and depth maps unified as 480 x 640. Particularly, 795 image-depth pairs are used to train the RGB-D model, and the remaining 654 utilized for testing. SUN-RGBD (Song et al., 2015) includes 10,335 RGB-D images with 530 x 730 resolution, where the objects are in 37 categories. All samples of this dataset divided into 5,285 and 5,050 splits for training and testing, respectively.
Implementation Details. Following SegNeXt Guo et al. (2022a), we employ Hamburger (Geng et al., 2021), a lightweight head, as the decoder to build our RGB-D semantic segmentation network. During finetuning, we only adopt two common data augmentation strategies, i.e., random horizontal flipping and random scaling (from 0.5 to 1.75). The training images are cropped and resized to 480 640 and 480 480 respectively for NYU Depthv2 and SUN-RGBD benchmarks. Cross- entropy loss is utilized as the optimization objective. We use AdamW (Kingma & Ba, 2015) as our optimizer with an initial learning rate of 6e-5 and the poly decay schedule. Weight decay is set to le-2. During testing, we employ mean Intersection over Union (mloU). which is averaged across semantic categories, as the primary evaluation metric to measure the segmentation performance. Following recent works (Zhang et al., 2023a; Wang et al., 2022; Zhang et al., 2023b), we adopt multi-scale (MS) flip inference strategies with scales (0.5, 0.75, 1, 1.25, 1.53.
C.3 RGB-D SALIENT OBJECT DETECTION
Datasets. We finetune and test DFormer on five popular RGB-D salient object detection datasets.
The finetuning dataset consists of 2, 2,195 samples, where 1,485 are from NJU2K-train (Ju et al., 2014) and the other 700 samples are from NLPR-train (Peng al., 2014). The model is evaluated on eight datasets, i.e., DES (Cheng et al., 2014) (135 samples), NLPR-test (Peng et al., 2014) (300), NJU2K-test (Ju et al., 2014) (500), STERE (Niu al., 2012) (1. 1000).. SIP (Fan et al., 2020) (929). The comparison of our method and other state-of-the-an art methods is shown in the Tab. 2.
Implementation Details. We set the output channel of Hamburger (Geng et al., 2021) head to 1, which is further added on the top of our RGB-D backbone to build the RGB-D salient object detection network. For model finetuning, we adopt the same data augmentation strategies and model optimizer in SPNet (Zhou et al., 2021a). For performance evaluation,
adopt four golden metrics of this task, i.e., cructure-measune (S) (Fan et al., 2017), mean absolute error (M) (Perazzi al., 2012), max F-measure (F) (Margolin et al., 201 4014), and max E-measure (E) (Fan et al., 2018).
20

---

Published as a conference paper at ICLR 2024
Stage
Stage
Stage
Stage
RGB
Layrr
RGB-D Block
H/s-W/
RGD D Black
RGB- D Block
H/16xW166
Consaleattoo
Decodar Head
H8
Depáh
Layyer
RGB
Prediction
xN1
xN2
xN3
xN4
RGB
PCL Steass
Stage 1
Stage
Stage 3
Stage 4
Lager
RGB D Block
H44xW44
RGB-D Block
19B W/B
Block
1V16x W/16
RGB-D Block
H332 W/32
Concirration
Docoder Head
H/8x W/B W/8
Deptts
Stam
Lagor
Prediction
x
N
xN3
xN4
Figure 13: Detailed illustration for the input features of the decoder head. Top: Only the features in the RGB branch are sent to the decoder. Bottom: The features of both RGB and depth branches are sent the decoder.
C.4 MORE DETAILS ABOUT THE DECODER INPUT FEATURES
Benefiting from the powerful RGB-D pretraining, the features of the RGB branch can better fuse the information of the two modalities compared to the ones with RGB pretraining. Thus, our decoder only takes the RGB features X instead of both X7 and X as input. The detailed structures of two forms are shown in Fig. 13.
D MORE VISUALIZATION RESULTS
this section, we provide more visualization results Fig. 14. Our DFormer produces higher segmentation accuracy than the current state-of-th the-art CMNext (MiT-B4). Moreover, the visual comparisons on RGB-D salient object detection are shown in Fig. 15.
E FUTURE WORK
We argue that there are number of foresccable directions for future research: 1) Applications on wider range of RGB-D downstream tasks. Considering that pretrained DFormer backbones own better RGB-D representation capabilities, it is promising to apply them to more RGB-D tasks, e.g., RGB-D face anti-spoofing (George & Marcel, 2021), 3D visual grounding (Liu et al., 2021a), RGB-D tracking (Yan et al., 2021), and human volumetric capture (Yu et al., 2021). 2) Extension other modalities. Based the well-designed framework, it is interesting produce multi-modal representations by substituting depth with other sensor data in other modalities, such as thermal and lidar data. The incorporation of additional modalities can facilitate the development of specialized and powerful encoders to meet the needs of various tasks. 3) More lightweight models. Current RGB-D methods are usually too heavy deploy to mobile devices. Although DFormer is efficient, there is still gap and development space in practical applications. We hope this paper could inspire researchers to develop better RGB-D encoding method, pushing the boundaries of what's feasible in real-world scenarios.
21

---

Published as a conference paper at ICLR 2024
Image
Depth
CMNext
Ours
Figure 14: Qualitative comparison of DFormer-L (Ours) and CMNext (MiT-B4) (Zhang et al., 2023b) on the NYU Depthv2 (Silberman et al., 2012) dataset.
22

---

Published as a conference paper at ICLR 2024
Image
Depth
GT
Ours
HiDANet
SPNet
Figure 15: Qualitative comparison of DFormer-l. (Ours), HiDANet (Wu et al., 2023), and SPNet Zhou et al. (2021a) the NJU2K dataset.
23


Reviews:
[{"review_id": "zNdu6ofMQ4", "reviewer": "Reviewer_HRuY", "summary": "In this paper, the authors propose an RGB-D pretraining framework for RGB-D semantic segmentation and salient object detection (SOD). First, they use an off-the-shelf depth estimator to generate depth maps for ImageNet-1K. Then, they use the image-depth pairs from ImageNet-1K to pretrain the backbone. Next, they insert an existing head on the backbone and then finetune the model on the RGB-D semantic segmentation and salient object detection datasets.", "questions": "Please see weaknesses.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1.\tTo improve the model performance, the authors pretrained the backbone on ImageNet-1K with image-depth pairs.\n2.\tThe authors conducted experiments on two RGB-D segmentation tasks.", "weaknesses": "1. The novelty and contributions are too limited. \nFirst, the proposed RGB-D block slightly modifies popular techniques, i.e., self-attention mechanism (Vaswani et al., 2017), depth-wise convolution, and attention weights (Hou et al., 2022), and combines them to fuse RGB and depth features. Second, the design of the RGB-D block follows the widely used idea in SOD, i.e., global and local information fusion. Third, the decoder directly uses the existing head from SegNext (Guo et al., 2022a) without any novel design. Thus, the contribution only comes from the pretraining idea, which is limited.\n\n2.\tThe authors missed some related methods [1-4] for comparison.\n\n[1] Visual Saliency Transformer. ICCV 2021.\n\n[2] 3-d convolutional neural networks for rgb-d salient object detection and beyond. TNNLS 2022.\n\n[3] Bi-Directional Progressive Guidance Network for RGB-D Salient Object Detection. TCSVT 2022.\n\n[4] UCTNet: Uncertainty-aware cross-modal transformer network for indoor RGB-D semantic segmentation. ECCV 2022.\n\n3.\tTo demonstrate the effectiveness of the pretrained backbone, the authors should replace the previous backbone in the compared methods with the proposed one to see whether improvements can be achieved.\n\n4.\tThe authors ignore existing pre-training methods [5, 6] for discussion and comparison.\n\n[5] RGB-based Semantic Segmentation Using Self-Supervised Depth Pre-Training\n\n[6] Self-Supervised Pretraining for RGB-D Salient Object Detection. AAAI 2022.\n\n5.\tSome widely used RGB-D SOD benchmark datasets [7-9] are also ignored.\n\n[7] Depth-induced multi-scale recurrent attention network for saliency detection. ICCV 2019.\n\n[8] Learning selective mutual attention and contrast for rgb-d saliency detection. TPAMI 2021.\n\n[9] Saliency detection on light field. CVPR 2014.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zmx0FSoyB5", "reviewer": "Reviewer_zqyp", "summary": "The paper introduces a RGB-D pretraining framework for transferable representations in RGB-Depth segmentation tasks. In the proposed methods DFormer, the RGB-depth backbone is pretrained using RGB-D pairs from ImageNet-1K, with the aim of enabling effective encoding of RGB-D information. It incorporates a sequence of RGB-D blocks designed for optimal representation of both RGB and depth data.", "questions": "What is the effect of using different depth estimation models? How effective is the accuracy of depth estimation for RGBD model pre-training, and will there be accumulation of errors?\n\nHow is the comparison between the fusion building block and the fusion module proposed in previous methods, such as cmx? Also, do the authors try to perform RGB-D pretraining for other methods, so as to perform a more comparable setting? \n\nHow does the DFormer perform if only RGB pretrain + D initialization for finetuning?\n\nHow is the effect and how is improvement from the light hamburger decoder in the proposed model? Whether the authors try to use other decoders? \n\nWhy to perform feature interaction between RGB and depth information in the last two stages?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed RGB-D pretraining framework can be used to solve the representation distribution shift between RGB and the depth information, and to increase the performance of RGB-D representation. \n\nA building block is proposed to perform RGB and depth feature interaction early in the pretraining stage, thus it is possible to reduce the interaction outside the backbone in the fine-tuning stage.", "weaknesses": "The comparison of using RGB-Depth pretraining on other previous works is missing. The most improvement seems from the join pretraining by using additional depth information as compared to previous methods.\n\nThe analysis of the depth generation is less included. Only one depth estimation method is used to generate the depth image for ImageNet. \n\nThere is generalization limitation in combining two modalities for pre-training. The performance of pre-training or fine-tuning on downstream tasks seems to be highly dependent on the generation or estimation of another modality besides RGB.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wGmSu47wTW", "reviewer": "Reviewer_KRMX", "summary": "This paper presents an RGB-D scene understanding framework with RGB-D pertaining weights. Two tasks are considered, including RGB-D semantic segmentation and salient object detection. A global awareness attention module and a local enhancement attention module are designed. RGB-D pre-training is performed on ImageNet-1K with estimated depth data. The proposed model achieves state-of-the-art performance and maintains good efficiency compared to existing works. As shown in Table 3, the benefit of using RGB-D pre-training is significant. Extensive ablation studies and parameter studies are conducted.", "questions": "The proposed model is highly efficient and it has large gains thanks to the RGB-D pertaining.  If the RGB-D pertaining strategy is applied to heavier state-of-the-art RGBD segmentation models like CMX and CMNeXt, how much gain can be achieved? If it is possible, this could be assessed and would help provide a fairer comparison.\n\nFig. 11 shows that the proposed module is sophisticated. Would it be nice to provide more detailed ablations to study other design choices based on this module architecture?\n\nSincerely,", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work is one of the first works to consider RGB-D pertaining to enhance RGB-D scene understanding. The results show that the benefit of RGB-D pretraining is significant.\n2. The proposed model is highly efficient compared to existing works.\n3. Table 1 presents a nice way of comparing with code-public-available works.\n4. The paper is overall well-written and nicely structured.", "weaknesses": "1. MultiMAE also uses RGB-D pretraining. However, in this work, a different depth estimation model is used. Would it be nice to provide a more fair comparison by using the same depth estimation model as MultiMAE to produce ImageNet depth data?  \n2. Again regarding fairness, the RGB-D pertaining is based on ImaegNet RGB-D data, and the depth estimation leverages important knowledge learned on other datasets. However, this knowledge is not used by existing RGB-D segmentation works like CMX. This can be discussed.\n3. Will the pretraining weights be released? Would the ImageNet depth data be released? This could be discussed.\n4. In the introduction, it was argued: \"the interactions are densely performed between the RGB branch and depth branch during\nfinetuning, which may destroy the representation distribution\". Do you have any observations to support this argument? E.g., some destroyed distributions or feature maps could be analyzed.\n5. There are still some writing mistakes. E.g., \"we conduct a depth-wise convolution\" should be \"We conduct a depth-wise convolution\". \"Our DFormer perform better segmentation accuracy than the current state-of-the-art\" should be \"Our DFormer produces higher segmentation accuracy than the current state-of-the-art\". \n6. ACNet (ICIP 2019) should be added to Table 1.\n7. How to scale to other modalities like RGB-thermal, RGB-LiDAR, X-Y-Z data, or even more modalities and datasets? This is not well discussed in the future work section. Different from depth data which can be produced by robust depth estimation models, it is harder to have large-scale thermal and LiDAR datasets for pertaining. This can be better discussed.\n8. As the main contribution lies in the study of RGB-D pertaining, more and recent advanced pertaining strategies could be compared and discussed. The main technical design lies in the fusion blocks, but there are no specific pertaining designs. Please discuss this and assess more pertaining choices.\n\nSincerely,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)