MonoMAE: Enhancing Monocular 3D Detection through Depth-Aware Masked Autoencoders
Authors: Xueying Jiang
Sheng Jin
Xiaoqin Zhang
Ling Shao
Shijian Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: Monocular 3D Object Detection
Masked Autoencoders
Primary area: machine_vision
TL;DR: This paper presents MonoMAE, a monocular 3D detector inspired by Masked Autoencoders that addresses the object occlusion issue in monocular 3D object detection by masking and reconstructing objects in the feature space.

Abstract: Monocular 3D object detection aims for precise 3D localization and identification of objects from a single-view image. Despite its recent progress, it often struggles while handling pervasive object occlusions that tend to complicate and degrade the prediction of object dimensions, depths, and orientations. We design MonoMAE, a monocular 3D detector inspired by Masked Autoencoders that addresses the object occlusion issue by masking and reconstructing objects in the feature space. MonoMAE consists of two novel designs. The first is depth-aware masking that selectively masks certain parts of non-occluded object queries in the feature space for simulating occluded object queries for network training. It masks non-occluded object queries by balancing the masked and preserved query portions adaptively according to the depth information. The second is lightweight query completion that works with the depth-aware masking to learn to reconstruct and complete the masked object queries. With the proposed feature-space occlusion and completion, MonoMAE learns enriched 3D representations that achieve superior monocular 3D detection performance qualitatively and quantitatively for both occluded and non-occluded objects. Additionally, MonoMAE learns generalizable representations that can work well in new domains.

MonoMAE: Enhancing Monocular 3D Detection
through Depth-Aware Masked Autoencoders
Xueying Jiang', Sheng Jin', Xiaoqin Zhang", Ling Shao Shijian Lu
SSLaab Nanyang Technological University, Singapore 2 College of Computer Science and Technology. Zhejiang University of Technology, China LUCAS-Terminne Al Lab, University Chinese Academy of Sciences, China
Abstract
Monocular 3D object detection aims for precise 3D localization and identification of objects from a single-view image. Despite its recent progress, it often struggles while handling pervasive object occlusions that tend to complicate and degrade the prediction of object dimensions, depths, and orientations. We design MonoMAE, a monocular 3D detector inspired by Masked Autoencoders that addresses the object occlusion issue by masking and reconstructing objects in the feature space. MonoMAE consists of two novel designs. The first is depth-aware masking that selectively masks certain parts of non-occluded object queries in the feature space for simulating occluded object queries for network training. It masks non-occio non-occluded object queries by balancing the masked and preserved query portions adaptively according to the depth information. The second is lightweight query completion that works with the depth- masked object queries. With the proposed feature-space occlusion and completion, MonoMAE learns enriched 3D representations that achieve superior monocular 3D detection performance qualitatively and quantitatively for both occluded and non- occluded objects. Additionally, MonoMAE learns generalizable representations that can work well in new domains.
masking to learn to reconstruct and complete the
Introduction
3D object detection has emerged as one key component in various navigation tasks such as au- tonomous driving, robot patrolling, etc. Compared with prior studies relying on LiDAR (69 25 (44 or multi-view images 227 33 61]. monocular 3D object detection offers more cost-effective and accessible alternative which identifies objects and predicts their 3D locations from single-view images. On the other hand, monocular 3D object detection is much more challenging due to the lack of 3D information from multi-view images or LiDAR data.
Among various new challenges in monocular 3D detection, object occlusion, which exists widely in natural images as illustrated in Figure (a). becomes a critical issue while predicting 3D locations in terms of object depths, object dimensions, and object orientations. Most existing monocular 3D detectors such as MonoDETR demonstrates clear performance degradation as illustrated in Figure (b). A simple idea is to learn to reconstruct the occluded object regions whereby occluded objects can be handled similarly as non-occluded objects. On the other hand, reconstructing occluded object regions in the image space complicated due to the super-rich variation of object occlusions in scene images.
and GUPNet [7 neglect the object occlusion issue which
Inspired by the Masked Autoencoders (MAE) [15] that randomly occludes image patches and reconstructs them in representation learning, we treat object occlusions natural masking and train
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

62% (17725) cars in KITTI 3D are occluded:
GUPNet MonoDETR
Red boves:
Occluded Cars
boxes: Non Occluded Cars
25.0
Ours
Performance
15.0
10.0
Occluded
Non-Occluded
BEV Occluded
Non-Occluded
Figure 1: Object occlusion is pervasive and affects monocular 3D detection: Object occlusion is pervasive, e.g., 62% (17725) cars in the KITTI 3D dataset suffer from various occlusions as illustrated in (a). Prevalent monocular 3D detection techniques such as GUPNet 37 and MonoDETR (66 are clearly affected by object occlusions in both 3D space (3D) and the bird's eye view (BEV) space as in (b). The proposed MonoMAE simulates and learns object occlusions by feature masking and completing which improves detection consistently for both occluded and non-occluded objects.
networks complete occluded object regions to learn occlusion-toleran representations. To this end, we design MonoMAE, a novel monocular 3D detection framework that adopts the idea of MAE by first masking certain object regions in the feature space (for simulating object occlusions) and then reconstructing the masked object features (for learning ceellusoonttlleraa representations). MonoMAE consists of depth-aware masking module and lightweight completion network. The depth-aware masking simulates object occlusions by masking the features of non-occluded objects adaptively according to the object depth information. It generates pairs of accloccldded and masked (i.e., occluded) object representations that can be directly applied to train the lightweight completion network, aiming for completing the occluded objects and learning aeclusion-toleraat representations. Note that MonoMAE introduces little computational overhead in inference time it requires no object masking in the inference stage.
The contributions of this work can be summarized in three major aspects. First, we design MonoMAE, a MAE-insppree monocular 3D detection framework that tackles object ocelusions effectively by masking and reconstructing object regions in the feature space. To the best of our knowledge, this is the first work that explores masking-recomm reaksttggcco for the task of monocular 3D object detection. Second, we design adaptive image masking and lightweight completion network that mask non- occluded objects adaptively according to the object depth (for simulating object occlusions) and reconstruct the masked object regions (for learning occlusion-toleran representations), respectively. Third, extensive experiments over KITTI 3D and nuScenes show that MonoMAE outperforms the state-of-the-ar consistently and it can generalize to new domains well.
2 Related Work
2.1 Monocular 3D Object Detection
Monocular 3D detection aims for the identification and 3D localization of objects from single-
view image. Most existing work can be broadly classified into two categories. The first employs
convolutional neural networks, where most methods follow conventional 2D detectors 12
23
The
standard approach learns monocular 3D detectors from single-view images only [7
8
70
66
34]
In addition, several studies explore leverage extra training data, such as LiDAR point clouds 39
55 6 48 46 45), depth maps [1 47 45
to acquire more depth information. Beyond that, several studies exploit the geometry relation between 2D and 3D spaces in different ways. For example, M3D-RPN [1 applies the powerful 2D detector FPN [49 for 3D detection. MonoDLE 40) aligns the centers of 2D and 3D boxes for better 3D localization.
22 38) and 3D CAD models 7 35 42
GUPNet [37] leverages uncertainty modeling to estimate the height of 3D boxes from the 2D boxes.

---

The second introduces powerful visual transformers 722
for more accurate monocular 57 56]. For example, MonoDTR
21 2 4
65
3D detection 19
66
7
19 integrates context- and depth-
aware features and injects depth positional hints into transformers. MonoDETR [66] modifies the transformer be depth-aware and guides the detection process by contextual depth cues. However, most existing methods neglect object occlusions that exist widely in natural images and often degrade the performance monocular 3D object detection clearly. We adopt the transformer architecture to learn ocelusion-tolerant representations that can handle object occlusion effectively without requiring any extra training data or annotations.
2.2 Occlusions in 3D Object Detection
Object occlusion is pervasive in scene images and it has been investigated in several 2D and 3D vision tasks [63 52
10
28
26
29
301 One typical approach learns to estimate the complete
localization of occluded objects. For example, Mono-3DT [18] estimates complete 3D bounding boxes by re-identifying occluded vehicles from a sequence of 2D images. BtcDet [59] leverages object shape priors to learns to estimate the complete shapes of partially occluded objects. Several studies consider the degree of occlusions in training. For example, MonoPair [8] exploits the relation of paired samples and encodes spatial constraints of occluded objects from their neighbors. HMF (3] introduces an anti-occlusion loss to focus on occluded samples. Different from existing methods, the proposed MonoMAE learns enriched and occlusion- representations by masking and completing object parts in the feature space.
2.3 Masked Autoencoders in 3D Tasks
Masked Autoencoders (MAE) [15] learn visual representations by masking image patches and reconstructing them, and it has been explored in several point cloud pre-training studies. For outdoor point cloud pre-training, Occupancy-MAE 41 exploits range-aware random masking that employs three masking levels to deal with the sparse voxel occupancy structures of LiDAR point clouds. GD-MAE (62) introduces a Generative Decoder to merge the surrounding context to restore the masked tokens hierarchically. For indoor point cloud pre-training, Point-MAE 43 adopts MAE to directly reconstruct the 3D coordinates of masked tokens. 12P-MAE (67] introduces 2D pre-trained models, and it enhances 3D pre-training with diverse 2D semanties. PiMAE (5] learns cross-modal representations with MAE by interactively handling point clouds and RGB images. Different from existing studies, the proposed MonoMAE handles monocular 3D detection from single-view images and it focuses on object occlusions by learning to complete occluded object regions in the feature
space.
Proposed Method
3.1 Problem Definition
Monocular 3D detection takes single RGB image as input, aiming to classify objects and predict their 3D bounding boxes. The prediction of each object composed of the object category C, a 2D bounding box BRD, and a 3D bounding box BaD. The 3D bounding box B3D can be decomposed to the object 3D location (æan, Mar, z3p), the object dimensions in object height, width and length (hap, wan, lan), as well as the object orientation 0.
3.2 Overall Framework
Figure 2 shows the framework of the proposed MonoMAE. Given an input image I, the 3D Backbone
first generates sequence of 3D object queries Q= [q 82.- . 99x (K denotes query number), and
the Non-Occluded Query Grouping then classifies the queries into two groups including non-occluded queries Q:O No 1 are the number of non-oceluded and occluded queries). The Non-Occluded Query Masking then masks Qo produce masked queries according their depth D [d,- d2., , dyl, leading to the masked queries QM M M ..... ........ M1. The Query Completion further reconstructs Q.M to produce the completed queries Q lof.
No and occluded queries Qo [ Φ - (U and V
g Finally, the occluded queries Qº and the completed queries QC are concatenated and fed to the Monocular 3D Detection for 3D detection
3

---

Non-Oeeluded Query Misking
Query Completion
quarry query
Depth Aware Masking
M
Querris
query
query
query
query 0
query Musked Queries Qu
Completion Natork
query Completed Queriies Qc
Non-Occluded
quary
Quaries Que
Impui
3D Backbone
query k quary Sequarioc of 3D Object Queries
Nun Occluded Query Grouping
Image!
query
Monocular Detection
3D Query
Non-Occhudel Query
query
2 Masked Query
Complesed Query
query V
3D Desection Heal
3D Resats
Cuncstanat
Occludal Query
Queries Qo
The framework of MonoMAE training: Given a single-view image, the 3D Backbone extracts 3D object query features which are grouped into non-occluded query features and occluded query features by the Occludec Query Grouping. The Depth- Aware Masking then masks the non-occluded query features to simulate object occlusions adaptively based the object depth, and the Completion Network then learns to reconstruct the masked queries. Finally, the completed and the occluded query features concatenated to train the 3D Detection Head for 3D predictions.
Figure 2:
predictions. Note the inference does not involve the Non-Occluded Query Masking, and it just concatenates the completion of occluded queries Qo (i.e., QC) with the non-occluded queries Q and feeds the concatenated queries to the 3D Detection Head for 3D predictions.
3.3 Non-Occluded Query Masking
Queries predicted by the 3D Backbone are either occluded or non-occluded, depending on whether the corresponding objects are occluded in the input image. In MonoMAE, mask the non-occluded queries in the feature space to simulate occlusions, aiming to generate pairs of non-occluded and masked (i.e., occluded) queries for learning occlusion-tolerant object representations.
Specifically, we design a Non-Occluded Query Grouping module identify non-occluded queries and then feed them into Depth- Aware Masking module to synthesize occlusions, with more detail be elaborated in the following subsections.
Non-Occluded Query Grouping. The Non-Occluded Query Grouping classifies the queries based on whether their corresponding objects occluded non-oceluded. With no information about whether the input queries are occluded, we design an occlusion classification network Po to predict the occlusion conditions OP = [of og; ..., [x] of queries Q 94::92: ... 9g.) where for the i-th query of = Po():). The Non-Occluded Query Grouping can be formulated by:
qi E QO {
if of = if of =
qi E Q
where of = O denotes the query is non-occluded, and of 1 denotes the query is occluded. The occlusion classification network trained with the occlusion classification loss Love as follows:
Loce = CE(OP, O**),
where CE is the Cross Entropy loss. We adopted the bipartite matching to match the predicted queries and objects in the image, where only matched queries have ground truth Ost of KITTI about whether they are occluded or not.
3D 13
Depth-Aware Masking. We design aepth- masking to adaptively mask non-occluded query features to simulate occlusions in the feature space, aiming to create occluded and occluded (i.e.. masked) pairs for learning occlusion-toled tolerann representations. As illustrated in Figure 3 the depth-aware masking determines
mask ratio according to the object depth the closer the object, the larger the mask ratio, thereby compensating the information deficiency of distant objects. In addition, we simulate occlusions by masking the feature space, as masking and reconstructing at the image level is complicated and computationally intensive.

---

Depth
Mask Ratão
Near
Large
Depth-Awar Masking
Small
Original Features the Query
Masked Features in the Query
Figure 3: Illustration of the Depth-Aware Masking. (a) Objects farther away usually smaller capturing less visual information. (b) The Depth-Aware Masking determines the mask ratio of an object according its depth- the closer the object is, the larger the mask ratio is applied, thereby compensating the information deficiency for objects that have larger distances from the camera.
The depth-aware masking first obtains the query depth before query masking. Without backward gradient propagation, it adopts the 3D Detection Head to obtain the depth D [d, dz. ... dyy for non-occluded queries. With the predicted depth, each non-occluded query is randomly masked as illustrated in Figure Specifically, objects that are more distant from the camera are usually captured with less visual information. The depth-aware masking accommodates this by assigning a smaller masking ratio to them, thereby keeping more visual information for distant objects for proper visual representation learning.
The mask ratio of each query is determined by:
1.0 d:/ Dmaa:
where r is the applied mask ratio for each query, d: is the depth for the i-th query, and Dmaz is the maximum depth in datasets. The masks M [m,, My: ,,yl generated for queries obey Bernoulli Distribution.
Finally, the query masking is formulated by:
M NO gi
m4
(4)
where g is the masked query, NoO is the non-occluded query, and m, is the generated mask.
3.4
Query Completion
The query completion learns reconstruct the adaptively masked queries, aiming to produce com- pleted queries whereby the network learns occlusion-toleran representations that are helpful in detecting occluded objects. We design a completion network &cc to reconstruct the masked queries. The Completion Network has an hourglass structure consisting of three conv-bn-relu blocks and one conv-bn block for 3D query completion. The completed query qf is obtained by:
(5)
where gf is the masked query. The Completion Network is trained under the supervision of the non-occluded queries before masking, where a completion loss Loom is formulated follows:
Leom NO Q²),
(6)
where I denotes the 1moothL1 loss [14, NO denotes the non-occluded queries, and Q denotes the queries completed by the Completion Network.
3.5 Loss Functions
The overall objective consists of three losses including Troces Treoms and Tobase where Teace and Trcom defined in Equation 2 and Equation 6 and Lbose denote losses for supervising the 3D box
5

---

Table Benchmarking on the KITTI 3D test set. All experiments adopt AP Ruo threshold of 0.7. Best in bold, second underlined.
metric with an IoU
Method
Venue
Extra Data
APan(IoU= 0.7)
Rao
AP BPev (loU= 0.7)
Ruo Hard
Easy
Moderate
Hard
Easy
Moderate
MonoRUn 6
CVPR 21
19.65
12.30
10.5827.94
17.34
15.24
MonoDTR 19
CVPR 22
21.99
15.39
12.73 28.59
20.38
17.14
MonoDistill 9
ICLR 22
LiDAR
22.97
16.03
13.6031.87
22.59
19.72
DID-M3D 45
ECCV 22
24.40
16.29
13.75
32.95
22.76
19.83
MonoNeRD [58
ICCV 23
22.75
17.13
15.63
31.13
23.46
20.97
D4LCNN 11
CVPR 20
16.65
11.72
9.51
22.51
16.02
12.55
DDMP-3D 153
CVPR 21
Depth
19.71
12.78
9.80
28.08
17.89
13.44
DD3D
ICCV 21
23.22
16.34
14.20
30.98
22.56
20.03
Kinematic3D 2
ECCV 20
Video
19.07
12.72
9.17
26.69
17.52
13.10
AutoShape 335
ICCV 21
CAD
22.47
14.17
31.36,0.66
20.08
15.59
MonoFlex 68
CVPR
19.94
13.89
12.07
28.23
19.75
16.89
MonoRCNN [50
ICCV 21
18.36
12.65
10.03
25.48
18.11
14.10
GUPNet 37
ICCV 21
20.11
14.20
11.77
DEVIANT 24
ECCV 22
21.88
14.46
11.89
29.65
20.44
17.43
MonoCon 32
AAAI 22
None
22.50
16.46
13.95 31.12
22.10
19.00
MonoDETR (66
ICCV 23
25.00
16.47
13.58
33.60
22.11
18.60
MonoUNI 20
NeurIPS 23
24.75
16.73
13.49
MonoCD 660
CVPR 24
25.53
16.59
14.53
33.41
22.81
19.57
Ours
None
25.60
18.84
16.78
34.15
24.93
21.76
Specifically, Thase includes losses for supervising the 3D box predictions including each object's 3D locations, height, width, length and orientation. We set the weight for each loss item to 1.0, and the overall loss function is formulated follows:
predictions.
L == Loce + Leon + Libase
Experiments
4.1 Experimental Settings
Datasets. We benchmark our method over two public datasets in monocular 3D object detection.
KITTI 3D [13 comprises 7,481 training images and 7,518 testing images, with training-data labels publicly available and test-data labels stored on a test server for evaluation. Following 77]. we divide the 7,481 training samples into new train set with 3,712 images and validation set with 3,769 images for ablation studies.
comprises 1,000 video scenes, including RGB images captured by 6 surround-vien cameras. The dataset split into a training set (700 scenes), a validation set (150 scenes), and test set (150 scenes). Following reported.
NuScenes 3
50 37 24 20 the performance on the validation set of nuScenes is
In addition, we perform evaluations on the most representative Car category of KITTI 3D and nuScenes datasets as in prior studies [51] 50 54 66
Evaluation Metrics. For KITTI 3D. we follow [5]] and adopt R the average of the AP of 40 recall points as the evaluation metric. We report the average precision BEV and 3D object detection by APBEV [R, and APa Rao with a threshold 0.7 for both test and validation sets. For the nuScenes dataset, adopt the mean absolute depth errors [00] in evaluations.
Implementation Details. We conduct experiments on one NVIDIA V100 GPU and train framework for 200 epochs with batch size of 16 and a learning rate of 2 x 10-1, We use the AdamW (36] optimizer with weight decay 10-4 We employ ResNet-50 [66] as the Transformer- based backbone and adopt the 3D detection head from [66] our detection framework.

---

Ground Truth
GUPNet
MonoDETR
Outs
Cisc
Clocc
Figure 4: Detection visualization over the KITTI val set. Ground truuh annotations are highlighted by red boxes, and predictions by MonoMAE and two state-of oftte-o--the-eeee green boxes. Red arrows highlight objects that have very different predictions across the compared methods. The ground truth LiDAR point clouds is provided for visualization only, and they are not used in MonoMAE training. Best viewed in color and zoom-in.
the-art art methods are highlighted by
Ablation study of technical designs in MonoMAE on the KITTI 3D val set. 'NOQG', 'DAM', and 'CN' denote Non-Occluded Query Grouping, Depth-Aware Masking, and Completion Network, respectively. The symbol indicates the baseline. The best results are highlighted in bold.
Table 2:
AP3D O.7)
AE'BEV 0.7) Rao
Index NOQG DAM CN
Easy
Moderate Hard
Easy
Moderate
Hard
1
24.85
16.21
14.74
34.53
23.99
18.84
23.33
15.09
13.20
32.68
21.80
17.43
27.33
18.52
14.95
36.51
24.21
19.12
V
/
24.69
15.71
13.57
33.46
23.03
18.19
V
v
27.25
18.76
15.45
36.81
25.18
20.05
V
28.39
19.35
15.87
37.59
26.27
21.33
30.29
20.90
17.61
40.26
27.08
23.14
4.2 Benchmarking with the State-of-the-Art
We benchmark MonoMAE with state-of-the-an monocular 3D object detection methods both quanti- tatively and qualitatively.
Quantitative Benchmarking. Table shows quantitative experiments on the test set of dataset KITTI 3D, where all evaluations were performed on the official online test server [13] for fairness. We can see that MonoMAE achieves superior detection performance consistently across all metrics, without using any extra training data such as image depths, video sequences, LiDAR points, and CAD models. addition, MonoMAE outperforms more the Moderate and Hard categories where various occlusions happen much more frequently than the Easy category. The superior performance largely attributed to our designed depth-aware masking and completion network, which masks queries simulate object occlusions in the feature space and reconstructs the masked queries to learn occlusion-tolens -tlleraat visual representations, respectively.
Qualitative Benchmarking. Figure shows qualitative benchmarking the KITTI 3D val set. be observed that compared with two state-of of-the-ar methods GUPNet and MonoDETR, the proposed MonoMAE produces more accurate 3D detection consistently for both non-occluded and occluded objects, even for challenging scenarios like distant objects. Specifically, GUPNet and MonoDETR tend to miss the detection of highly occluded object in Cases I and 2 as highlighted by arrows. a comparison, MonoMAE performs clearly better by detecting those challenging objects successfully, demonstrating its superior capability on handling object occlusions.

---

Table 3:
Ablation study of masking strategies on the KITTI 3D val set. The best results are in bold.
0.7) R4o
APBey(loU 0.7) Rao
Index
Masking Strategy
Easy
Moderate
Hard
Easy Moderate
Hard
Image Masking
20.51
15.03
13.74
19.74
16.71
2
Query Masking (w/o Depth-Aware)
27.14
18.47
15.02
36.98
25.52
20.64
Query Masking (w/ Depth-Aware)
30.29
20.90
17.61
40.26
27.08
23.14
Table 4:
Ablation study of the loss functions on the KITTI 3D val set. Loce and Loom refer the occlusion classification loss and the completion loss, respectively. The best results are in bold.
AP3D [Io== 0.7) Rio
AP BEv (loU= 0.7)
Index
Loce
Leom
Easy
Moderate
Hard
Easy
Moderate
Hard
28.37
19.61
16.01
37.48
26.55
21.50
2
v
26.36
19.15
15.88
36.76
26.49
22.62
3
/
v
30.29
20.90
17.61
40.26
27.08
23.14
4.3 Ablation Study
We conduct extensive ablation studies to examine the proposed MonoMAE. Specifically, we examine MonoMAE from the aspect of the technical designs, query masking strategies, as well as loss functions.
Network Designs. We examine the effectiveness of two key designs in MonoMAE, namely, the Depth-Aware Masking module (DAM) and the Completion Network (CN) (on the validation set of KITTI 3D), as shown in Table 2 We formulate the baseline by including the Non-Occluded Query Grouping module (NOQG), which does not affect the network training as both identified occluded and non-occluded queries are fed train 3D detectors. When CN is not used in Rows 2 and 4. the 3D detection degrades as queries are masked but not reconstructed which leads to further information loss. While not incorporating DAM in Rows and 5, the detection improves clearly compared with the baseline, as the completion helps learn better representations for naturally occluded queries. In addition, incorporating DAM and CN on top of NOQG Row 7 performs clearly better than incorporating DAM and CN alone in Row 6, as the former applies masking and completion to non-oceluded queries only. also shows that masking naturally occluded queries to train the completion network is harmful to the leamed representations.
Masking Strategies. We examine how different masking strategies affect monocular 3D detection. We studied three masking strategies as shown in Table The first strategy masks the input images randomly, aiming assess the value of masking and completing the feature instead of image space, We can observe that the image-level masking yields clearly lower performance as compared with query masking in the feature space, largely due to the complication in masking and reconstructing images with lightweight completion network. The second strategy masks query features randomly without considering object depths, aiming to evaluate the importance of object depths in query masking. The experiments show that random query masking outperforms the image-level masking significantly. The third strategy performs the proposed depth-aware query masking. It outperforms the feature-space random masking consistently, demonstrating the value of object depths for query masking.
Loss Functions. We examine the impact of the occlusion classification loss Tooe and the completion loss Leom in Equations and [ where Loce supervises the occlusion classification network (in Non-Occluded Query Grouping) to predict whether the queries are occluded and Leom supervises the Completion Network reconstruct the masked queries. As Table T shows, while implementing Loec alone, the occlusion prediction is supervised while the query reconstruction is unsupervised. The network under such an objective does not learn well as the Completion Network cannot reconstruct object queries well without sufficient supervision. While implementing Leom alone, the occlusion classification network cannot identify occluded and non-occluded queries accurately where many occluded queries are fed for masking, leading to more query occlusion and poor detection performance. While employing both losses concurrently, the performance improves significantly as non-occluded queries can be identified for masking and reconstruction, leading to peclusion-tolerane representations.

---

Table 5: Comparison inference speed of several monocular 3D detection methods. Ours denotes the proposed MonoMAE without including the Completion Network.
Method Inference Time (ms)
GUPNet [37]
MonoDTR
[19] MonoDETR
[66] Ours*
Ours
40
37
36
38
Table 6:
Cross-dataset evaluations that perform training on the KITTI train set, and testing on the KITTI val and nuScenes val sets. We adopt the evaluation metric mean absolute error of the depth ()). Best is highlighted in bold, and second underlined.
KITTI Val
nuScenes frontal Val
Method
0-20
20-40
40-oc
All
0-20
20-40
40-oc
All
M3D-RPP RED-RPN 1
0.56
1.33
2.73
1.26
0.94
3.06
10.36
2.67
MonoRCNN [50
0.46
1.27
2.59
1.14
0.94
2.84
8.65
2.39
GUPNet 37
0.45
1.10
1.85
0.89
0.82
1.70
6.20
1.45
DEVIANT
0.40
1.09
1.80
0.87
0.76
1.60
4.50
1.26
MonoUNI 20
0.38
0.92
1.79
0.87
0.72
1.79
4.98
1.43
MonoMAE (Ours)
0.36
0.91
1.74
0.86
0.71
1.57
4.95
1.40
4.4 Discussions
Efficiency Comparison. We compare the inference time of several representative monocular 3D detection methods on the KITTI val set, where all compared methods are evaluated with one NVIDIA V100 GPU under the same computational environment for fairness. As Table 5 shows, GUPNet, MonODTR, and MOnODETR have an average inference time of 40ms, 37ms, and 43ms for each image, respectively. As a comparison, the proposed MonoMAE takes the shortest inference time, demonstrating its good efficiency in monocular 3D detection. Further, we analyzed the Completion Network in terms of network parameters and floating-pont point operations per second (FLOPs), showing it has very limited 2.22G parameters and 0.08M in FLOPs.
Generalization Ability. We examine the generalization capability of the proposed MonoMAE by directly applying the KITti-trained MonoMAE model to the car Category of the nuScenes validation set without additional training. The detection performance reported for reference. Table shows that MonoMAE attains the highest or second-highes detection performance across various metrics the nuScenes frontal validation set. This indicates that despite the domain shift from KITTI to nuScenes, MonoMAE still maintains satisfactory performance. Since DEVIANT 24 is equivariant to the depth translations, it sometimes has higher performance.
the KITTI validation set is also
Conclusion
This paper presents MonoMAE, novel method inspired by the Masked Autoencoders (MAE) to deal with the pervasive occlusion problem the monocular 3D object detection task. MonoMAE consists of two key designs. The first is a depth-aware masking module, which simulates the occlusion for non-occluded object queries the feature space during training. The second is a lightweight completion network, which reconstructs and completes the masked object queries. Quantitative and qualitative experiment results show that MonoMAE Icarns enhanced 3D representations and achieves superior monocular 3D detection performance for both occluded and non-occluded objects. Moving forward, we plan to investigate generative approaches to simulate natural occlusion patterns for various 3D detection tasks.
Limitations. MonoMAE leverages depth- masking to mask non-oceluded queries simulate object occlusions in the feature space. However, the masked queries may have different patterns as compared with the features of naturally occluded object queries. Such a gap could affect the reconstruction of complete queries and monocular 3D detection performance. This issue could be mitigated by introducing generative networks that learn distributions from extensive real-world data for generating occlusion patterns that are more similar to natural occlusions.

---

Acknowledgments and Disclosure of Funding
This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, well as cash and in-kind contribution from the industry partner(s).
References
[1] Garrick Brazil Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9287-9296, 2019.
[2] Garrick Brazil, Gerard Pons-Moll, Xiaoming Liu, and Bernt Schiele. Kinematic 3d object detection in monocular video. In Proceedings of the IEEE/CVF European Conference Computer Vision, pages 135 155-152.60 Springer, 2020.
[3] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11621-
11631, 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the IEEE/CVF European Conference on Computer Vision, pages 213-229. Springer, 2020.
Anthony Chen, Kevin Zhang, Renrui Zhang, Zihan Wang, Yuheng Lu, Yandong Guo, and Shanghang Zhang. Pimae: Point cloud and image interactive masked autoencoders for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5291-5301, 2023.
[6] Hansheng Chen, Yuyao Huang, Wei Tian, Zhong Gao, and Lu Xiong. Monorun: Monocular 3d object detection reconstruction and uncertainty propagation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10379-10388, 2021.
[7] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3d object detection for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2147 2156, 2016.
[8] Yongjian Chen, Lei Tai, Kai Sun, and Mingyang Monopair: Monocular 3d object detection using pairwise spatial relationships. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12093-12102, 2020.
[9] Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojic Li, Zhihui Wang, and Wanli Ouyang. Monodis- till: Learning spatial features monocular 3d object detection. International Conference on Learning Representations, 2022.
[10] Huazhen Chu, Lisha Mo, Rongquan Wang. Tianyu Hu, and Huimin Ma. Visibility of points: Mining occlusion cues for monocular 3d object detection. Neurocomputing, 502:48-56, 2022.
[11] Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi. Zhiwu Lu, and Ping Luo. Learning depth-guided convolutions monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1000-1001, 2020.
[12] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi. Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6569-6578, 2019.
[13] Andreas Geiger, Philip Lenzz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3354-3361. IEEE, 2012.
[14] Ross Girshick. Fast r-cna. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1440-1448, 2015.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (6000-16004 2022.
10

---

[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[17] Tong He and Stefano Soatto. Mono3d++: Monocular 3d vehicle detection with two-scale 3d hypotheses and task priors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8409 8416, 2019.
[18] Hou-Ning Hu, Qi-Zhi Cai, Dequan Wang. Ji Lin, Min Sun, Philipp Krahenbuhl, Trevor Darrell, and Fisher Yu. Joint monocular 3d vehicle detection and tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5390- 5399, 2019.
[19] Kuan- Chih Huang. Tsung- Han Wu, Hung-Time Ting Su, and Winston H Hsu. Monodtr: Monocular 3d object detection with depth-aware transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 4012-4021, 2022.
[20] Jinrang Jia, Zhenjia Li, and Yifeng Shi. Monouni: A unified vehicle and infrastructure-ssie monocular 3d object detection network with sufficient depth clues. In Advances in Neural Information Processing Systems, 2023.
[21] Xueying Jiang, Jiaxing Huang. Sheng Jin, and Shijian Domain generalization via balancing training difficulty and model capability. Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023.
[22] Xueying Jiang, Sheng Jin, Lewei Lu, Xiaoqin Zhang, and Shijian Lu. Weakly supervised monocular 3d detection with single-view image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[23] Sheng Jin, Xueying Jiang, Jiaxing Huang, Lewei Lu, and Shijian Lu. Lims meet vlms: Boast open vocabu- lary object detection with fine-grained descriptors. International Conference on Learning Representations, 2024.
[24] Abhinav Kumar, Garrick Brazil, Enrique Corona, Armin Parchami, and Xiaoming Liu. Deviant: Depth equivariant network for monocular 3d object detection. In Proceedings of the IEEE/CVF European Conference on Computer Vision, pages 664-683. Springer, 2022.
[25] Alex H Lang, Sourahh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12697-12705, 2019.
[26] Ke Li and Jitendra Malik. Amodal instance segmentation. In Proceedings of the IEEE/CVF European Conference on Computer Vision, pages 677-693. Springer, 2016.
[27] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learnigg bird's s-yce-view representation from multi-camera images via spatiotemporal trans- formers. Proceedings of the IEEE/CVF European Conference on Computer Vision, pages 1-18. Springer. 2022.
[28] Zhixuan Li, Weining Ye, Tingting Jiang, and Tiejun Huang. 2D amodal instance segmentation guided by 3D shape prior. In Proceedings of the IEEE/CVF European Conference on Computer Vision, pages 165 185-181,22 2022.
[29] Zhixuan Li, Weining Ye, Tingting Jiang, and Tiejun Huang. GIN: Generative invariant shape prior for amodal instance segmentation. IEEE Transactions on Multimedia, pages 3924-3936. 2023.
[30] Zhixuan Li, Weining Ye, Juan Terven, Zachary Bennett, Ying Zheng, Tingting Jiang, and Tiejun Huang. MUVA: A new large-scale benchmark for multi-view amodal instance segmentation in the shopping scenario. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23504- 23513, 2023.
[31] He Liu. Huaping Liu, Yikai Wang. Fuchun Sun, and Wenbing Huang. Fine-grained multilevel fusion for anti-occlusion monocular 3d object detection. IEEE Transactions on Image Processing, 31:4050-4061, 2022.
[32] Xianpeng Liu, Nan Xue, and Tianfu Wu. Learning auxiliary monocular contexts helps monocular 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1810-1818 2022.
11

---

[33] Yingfei Liu. Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In Proceedings of the IEEE/CVF European Conference on Computer Vision, pages 531-548. Springer, 2022.
[34] Zechen Liu, Zizhang Wu, and Roland Tóth. Smoke: Single-stage monocular 3d object detection via keypoint estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 996 90669997222 2020.
[35] Zongdai Liu, Dingfu Zhou, Feixiang Lu, Jin Fang, and Liangjun Zhang. Autoshape: Real-time shape-awan aware monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15641-15650, 2021.
[36] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, 2018.
[37] Yan Lu, Xinzhu Ma, Lei Yang. Tianzhu Zhang, Yating Liu, Qi Chu, Junjie Yan, and Wanli Ouyang. Geometry uncertainty projection network for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3111-3121, 2021.
[38] Xinzhu Ma, Shinan Liu, Zhiyi Xia, Hongwen Zhang, Xingyu Zeng, and Wanli Ouyang. Rethinking pscudo-lidar representation. Proceedings of the IEEE/CVF European Conference on Computer Vision, pages 31 1-3727 Springer, 2020.
[39] Xinzhu Ma, Zhihui Wang. Haojie Li, Pengbo Zhang. Wanli Ouyang, and Xin Fan. Accurate monocular 3d object detection via color-embedded 3d reconstruction for autonomous driving. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 6851-6860, 2019.
[40] Xinzhu Ma, Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang. Delving into localization errors for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4721 4730. 2021.
[41] Chen Min, Liang Xiao, Dawei Zhao, Yiming Nie, and Bin Dai. Occupaney-mac: Self-supervised pre- training large-scale lidar point clouds with masked occupancy autoencoders. IEEE Transactions on Intelligent Vehicles, 2023.
[42] Krishna Murthy, GV Sai Krishna, Falak Chhaya, and K Madhava Krishna. Reconstructing vehicles from single image: Shape priors for road scene understanding. In International Conference on Robotics and Automation, pages 724-731. IERF, 2017.
[43] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud sulf-supervised learning. In Proceedings of the IEEE/CVF European Conference on Computer Vision, pages 604 621. Springer, 2022.
[44] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular object detection? In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 3142-3152, 2021.
[45] Liang Peng, Xiaopei Wu, Zheng Yang, Haifeng Liu, and Deng Cai., Did-m3d: Decoupling instance depth for monocular 3d object detection. In Proceedings of the IEEF/CVF European Conference on Computer Vision, pages 71-88. Springer, 2022.
[46] Liang Peng, Junkai Xu, Haoran Cheng, Zheng Yang, Xiaopei Wu, Wei Qian, Wenxiao Wang, Boxi Wu, and Deng Cai. Learning occupancy for monocular 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[47] Zengyi Qin, Jinglu Wang, and Yan Lu. Monogrnet: A general framework for monocular 3d object detection. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(9):5170- 2021.
[48] Cody Reading, Ali Harakeh, Julia Chae, and Steven L. Waslander. Categorical depth distribution network for monocular object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8555 8564, 2021.
[49] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
[50] Xuepeng Shi, Qi Ye, Xiaozhi Chen, Chuangrong Chen, Zhixiang Chen, and Tae- Kyun Kim. Geometry- based distance decomposition for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15172 15181, 2021.
12

---

[51] Andrea Simonelli. Samuel Rota Bulo, Lorenzo Porzi, Manuel López-Aniequera, and Peter Kontschieder. Disentangling monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vixion, pages 1991-1999, 2019.
[52] Yongzhi Su, Yan Di, Guangyao Zhai, Fabian Manhardt, Jason Ramhach, Benjamin Busam, Didier Stricker, and Federico Tombari. Opa-3d: Ocelusion-aware pixel-wise aggregation for monocular 3d object detection. IEEE Robotics and Automation Letters, 8(3::1327-1334. 2023.
[53] Li Wang. Liang Du, Xiaoqing Ye, Yanwei Fu, Guodong Guo, Xiangyang Xue, Jianfeng Feng, and Li Zhang. Depth-conditioned dynamic message propagation for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 454-463, 2021.
[54] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Foosåd: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 913-922, 2021.
[55] Yan Wang. Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1445-8453, 2019.
[56] Zizhang Wu, Yuanzhu Gan, Lei Wang, Guilian Chen, and Jian Pu. Monopge: Monocular 3d object detection with pixel geometry contexts. International Conference on Robotics and Automation, 2023.
[57] Zizhang Wu, Yunzhe Wu, Jian Pu, Xianzhi Li, and Xiaoquan Wang. Atteation- based depth distillation with 3d-aware positional encoding for monocular 3d object detection. Proceedings of the AAAI Conference Artificial Intelligence, 2023.
[58] Junkai Xu, Liang Peng, Haoran Cheng, Hao Li, Wei Qian, Ke Li, Wenxiao Wang, and Deng Cai. Mononerd: Nerf like representations for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6814-6824, 2023.
[59] Qiangeng Xu, Yiqi Zhong, and Ulrich Neumann. Behind the curtain: Learning occluded shapes for 3d object detection. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2893 2901, 2022.
[60] Longfei Yan, Pei Yan. Shengzhou Xiong, Xuanyu Xiang, and Yihua Tan. Monocd: Monocular 3d object detection with complementary depths. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
[61] Chenyu Yang. Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird's-eye s-eye- view recognition via perspective supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 17839-17839. 2023.
[62] Honghui Yang, Tong He, Jiaheng Liu, Hua Chen, Boxi Wu, Binbin Lin, Xiaofei He, and Wanli Ouyang. Gd-mae: generative decoder for mae pre-training on lidar point clouds. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9403-9414, 2023.
[63] Hongdou Yao, Jun Chen, Zheng Wang, Xiao Wang, Pengfei Han, Xiaoyu Chai, and Yansheng Qiu. Occlusion-aware clane-constraints for monocular 3d object detection, IEEE Transactions on Intelligent Transportation Systems, 2023.
[64] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11784- 11793, 2021.
[65] Jingyi Zhang, Jiaxing Huang, Xucying Jiang, and Shijian Lu. Black-box unsupervised domain adaptation with bi-directional atkinson-shiie shiffrin memory. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11771-11782, 2023.
[66] Renrui Zhang, Han Qiu, Tai Wang, Xuanzhuo Xu, Ziyu Guo, Yu Qiao, Peng Gao, and Hongsheng Li. Monodetr: Depth-award transformer for monocular 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[67] Renrui Zhang, Liuhui Wang. Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d representations from 2d tre-trained models via amage-to-poing masked autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21769-21 -17800 2023.
13

---

[68] Yunpeng Zhang, Jiwen Lu, and Jie Zhou. Objects are different: Flexible monocular 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3289-3298,
2021.
[69] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4490-4499.
2018.
[70] Yunsong Zhou, Yuan He, Hongzi Zhu, Cheng Wang. Hongyang and Qinhong Jiang. Monoef: Extrinsic parameter free monocular 3d object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):10114-11 10128, 2021.
[71] Yunsong Zhou, Hongzi Zhu, Quan Liu, Shan Chang, and Minyi Guo. Monoatt: Online monocular 3d object detection with adaptive token transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2023.
[72] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. International Conference Learning Representations, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions are summarized at the end of the introduction section to accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The discussion of the limitations of the work is presented in Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- specifleciiii.nnnn asymptotic approximations only holding locally). The authors should reflect how these assumptions might he violated practice what the implications would be.
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when the image resolution is low images taken in low lighting. Or speech-to-texy to-tttt system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [NA]
Justification: The results provided in this paper are not theoretical, since the results are practical results tested on datascts.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- rofs-eeefeeeced.
15

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The datasets used for experiments and the implementation details are introduced in Section 4.1 to ensure the reproducibility of this work. Moreover, a detailed introduction to the architecture of the proposed approach is presented in Section of the paper and Section C.I of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g.. with open-sopnee sourc datasct instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer: [No]
Justification:
The used datascts are publicly available. We will consider releasing the code upon
acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines stttpe://niss.cc/pppt
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
16

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://nipa.ccpputee for more details.
Buzdee//DodeSubmiissioooPPo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should relcase anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all training and test details (e-g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are provided in the implementation details in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that necessary appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Following previous papers in the same field, the statistical significance is not provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It OK report I-sigma error hars, but one should state it. The authors should preferably report a 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric eиror bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
The information on the computer resources is provided in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn make into the paper).
9. Code Of Ethics
Question: Does research conducted the paper conform, every respect, with NeurIPS Code of Ethics htttp::/neuuri: http://eerriis.ccppaac/c//////aa
Answer: [Yes]
Justification:
This paper conforms to the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special considcration duc to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
foundational research not tied to particular
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
18

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The original papers of the used datasets are cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. www.coom//aaac /attttttstts has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-per re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
19

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
20


Reviews:
[{"review_id": "ybDgkZbpLi", "reviewer": "Reviewer_xT7k", "summary": "This paper introduces MonoMAE, a novel monocular 3D object detection framework designed to improve detection performance in the presence of object occlusions. MonoMAE leverages the concept of Masked Autoencoders, treating object occlusions as natural masking and training the network to complete occluded regions. This innovative approach addresses the pervasive issue of object occlusions in monocular 3D detection, leading to superior detection performance. Extensive experiments on datasets like KITTI 3D and nuScenes show that MonoMAE outperforms state-of-the-art methods in both qualitative and quantitative measures.", "questions": "1.  What is the accuracy of the occlusion classification network? How does the accuracy influence the whole 3D object detection network?\n2. What is the accuracy of the query completion network?", "limitations": "The authors discussed some failure cases in their paper, as well as the gap between the generated occlusion pattern and natural occlusion patterns.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of depth-aware masking to simulate occlusions and the use of a lightweight query completion network are innovative and address a significant challenge in monocular 3D detection.\n2. MonoMAE improves detection performance without the need for additional training data or annotations, making it a practical solution for real-world applications like autonomous driving and robotics.\n3. The framework demonstrates superior performance on benchmark datasets (KITTI 3D and nuScenes), outperforming existing state-of-the-art methods in both occluded and non-occluded scenarios.\n4. MonoMAE shows strong generalization capabilities to new domains, which is critical for deploying models in diverse environments.", "weaknesses": "1. In many datasets and methods, objects are not merely labeled as \"occluded\" or \"non-occluded.\" Instead, they may be assigned occlusion levels or degrees that quantify the extent to which an object is occluded. These levels provide more granularity and can influence how models are trained and evaluated. It would be beneficial to specify how occlusion levels are defined and used. Clarifying whether discrete or continuous levels are employed and how these influence the labeling, training, and evaluation processes will provide a clearer understanding of the methodology and its robustness in handling occlusions.\n2. The paper does not provide explicit details about the accuracy of the occlusion classification network or how this accuracy influences the overall 3D object detection network. This information appears to be missing.\n3. The paper does not explicitly report the performance or accuracy of the query completion network. Including a report on the performance of this network, such as quantitative results or visualization of the reconstructed queries, would be valuable. It would demonstrate whether the query completion network is learning meaningful features and contributing effectively to the overall 3D object detection performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.\n\n**Weakness 1: Occlusion Levels: Definition and Usage**\n\nWe appreciate the reviewers' insightful comments regarding the use of occlusion levels. Our method only uses the binary \\\"occluded\\\" and \\\"non-occluded\\\" labels. During training (Figure 2 of the submitted manuscript), the binary occlusion labels are used to train the Non-Occluded Query Grouping module for classifying the non-occluded queries that are masked and completed. During inference (Figure 5 of the submitted manuscript), the Non-Occluded Query Grouping module is used to classify the queries.\n\nTo this end, we only need to obtain the binary occlusion labels indicating whether the objects are occluded or not, without using the fine-grained occlusion labels which could make the task complicated. In this paper, we transform the ground truths of occlusion degrees $o$ provided in the KITTI 3D dataset, where $o \\in$ {0, 1, 2, 3}, into binary ground truths. The transformation simplifies $o=0$ as non-occlusion and $o \\in$ {1, 2, 3} as occlusion, leading to the binary ground truths of occlusion conditions $o^{gt} \\in$ {0, 1} with 0 indicating non-occlusion and 1 indicating occlusion.\n\n\n**Weakness 2 & Question 1: Accuracy and Impact of the Occlusion Classification Network**\n\nThank you for your suggestion. The accuracy of the occlusion classification network has been provided in Section F.3 of the submitted Appendix. The accuracy of the occlusion classification network is 96.46%, indicating that most object queries can be correctly classified as occluded or non-occluded.\n\nMoreover, to validate the influence of the classification accuracy on the overall 3D object detection network, we add experiments using a trained network with fixed weights. During each inference, the occlusion classification accuracy is manually adjusted to 50% and 70% with the help of ground-truth occlusion labels. The experimental results are shown in the table below, where the original accuracy (96.46%) of the occlusion classification network is also exhibited. The metrics $AP_{3D} (IoU=0.7)$ and $AP_{BEV}(IoU=0.7)$ with Easy, Moderate and Hard categories are used. In each cell of the table, the performance is listed as $AP_{3D}(IoU=0.7)$ / $AP_{BEV}(IoU=0.7)$.\n\n| Occlusion Classification Accuracy | Easy          | Moderate      | Hard          |\n|-----------------------------------|---------------|---------------|---------------|\n| 50%                               | 27.12 / 36.41 | 18.05 / 24.67 | 15.21 / 19.74 |\n| 70%                               | 28.47 / 37.82 | 19.51 / 26.90 | 16.02 / 21.85 |\n| 96.46%                            | **30.29 / 40.26** | **20.90 / 27.08** | **17.61 / 23.14** |\n\nFrom this table, we can observe that the higher occlusion classification accuracy can contribute to better 3D detection performance, validating the importance of classification accuracy on the overall 3D object detection network.\n\nWe will add these results and analyses in the revised version.\n\n\n\n**Weakness 3 & Question 2: Performance of the Query Completion Network**\n\nThank you for your suggestion! The performance of the Completion Network is measured through the similarity between the non-occluded queries before masking and the queries completed by the Completion Network. This similarity is defined by the Smooth L1 loss, as defined by Equation (6) of the submitted manuscript. \n\nWith the help of the Smooth L1 loss, we visualize the training losses with and without using the Completion Network in Figure 7 of the submitted Appendix. As Figure 7 shows, the training loss using the Completion Network (the orange line) drops while the training loss without using the Completion Network (the blue line) remains at a high level, demonstrating the effectiveness of the Completion Network in acquiring occlusion-tolerant representations effectively by learning to reconstruct the masked queries.\n\nThe visualization of masked and reconstructed queries is not provided since the queries are one-dimensional vectors and cannot be visualized in a meaningful approach.\n\nMoreover, Table 2 in the submitted manuscript can further validate the effectiveness of the Completion Network quantitatively. Comparing Rows 2 and 6, and Rows 4 and 7, using Completion Network can improve the 3D detection performance effectively, showing that the masked queries are properly reconstructed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WuSFpi1PaF", "reviewer": "Reviewer_8dEd", "summary": "This paper applies Masked Autoencoder to 3D object detection. It distinguishes object queries into occluded and non-occluded categories, and during training, it applies depth-aware masking to the non-occluded queries and learns by completing them. At test time, the completion is applied to the occluded queries.", "questions": "Please refer to the weaknesses.", "limitations": "Limitations are included in the main text.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It achieved state-of-the-art performance on the KITTI 3D dataset.\n- The idea of interpreting occluded queries as masked queries to solve the problem is interesting.\n- The training and test times are illustrated clearly in figures.", "weaknesses": "- As stated in the limitations section, occlusion at the image level and masking at the feature level of object queries are not the same. Further analysis is needed to understand the actual implications of masking in object queries.\n- If masking serves the role of occlusion at the image level, there should be no reason for the mask ratio to vary with depth, yet depth-aware masking is highly beneficial. An analysis is needed to understand why depth-aware masking works well compared to random masking.\n- In my opinion, the performance of the Non-Occluded Query Grouping classification is crucial for the framework to function properly. Although classification accuracy is provided in the supplementary material, it would be helpful to include various metrics such as precision, recall, and F1-score. If the results of the Non-Occluded Query Grouping classification are biased, it might be interesting to apply completion not only to the occluded queries but also to the non-occluded queries at test time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.\n\n\n**Weakness 1: Feature-Level Masking vs. Image-Level Masking**\n\nWe would clarify that the strategy of masking and completion aims to generate pairs of non-occluded and occluded (via masking) data for learning occlusion-tolerant object representations. The strategy could be implemented in either image space or feature space. We chose to mask and reconstruct query features, as masking and reconstructing images is complicated and computationally intensive due to noisy image data and super-rich occlusion patterns. As shown in Table 3 of the submitted manuscript, masking and completing query features perform clearly better than masking and completing images. Nevertheless, we understand that the simulated occlusion features are different from that of natural occlusions (as briefly discussed In the Limitations section), and shared that this issue could be mitigated by introducing generative networks to learn the distribution of natural occlusions.\n\n\n**Weakness 2: Depth-Aware Masking v.s. Random Masking**\n\nWe would clarify that the proposed depth-aware masking is implemented in the feature space. We examined how different masking strategies affect the monocular 3D detection by testing three masking strategies: 1) random masking in the image space; 2) random masking in the feature space; and 3) depth-aware masking in the feature space, as shown in Rows 1-3 in Table 3 of the submitted manuscript. We can observe that the depth-aware masking performs clearly the best. As discussed in Section A.1 of the submitted Appendix, the depth-aware masking modulates the mask ratio by lowering it for distant objects. This effectively retains more visual information for distant objects which are usually small and have limited pixels and visual information. In addition, it also facilitates the ensuing completion task as completing heavily masked small objects is challenging and liable to failures.\n\n\n**Weakness 3: Non-Occluded Query Grouping**\n\nThanks for your valuable suggestion! Below is the detailed performance of the Non-Occluded Query Grouping classification module with your suggested metrics.\n\n| Metrics     | Precision | Recall  | F1-Score | Accuracy |\n|-------------|-----------|---------|----------|----------|\n| Performance | 98.35 %   | 94.47 % | 96.38 %  | 96.46 %  |\n\nTo evaluate the performance when applying completion on all queries, both occluded and non-occluded, at the test time, we conduct experiments and show the results in the table below. The metrics $AP_{3D} (IoU=0.7)$ and $AP_{BEV}(IoU=0.7)$ with Easy, Moderate and Hard categories are used. In each cell of the table, the performance is listed as $AP_{3D}(IoU=0.7)$ / $AP_{BEV}(IoU=0.7)$.\n\n\n| Setting                           | Easy          | Moderate      | Hard          |\n|-----------------------------------|---------------|---------------|---------------|\n| Completing All Queries            | 28.46 / 37.17 | 18.40 / 25.95 | 15.23 / 21.82 |\n| Original                          | **30.29 / 40.26** | **20.90 / 27.08** | **17.61 / 23.14** |\n\nThe results indicate a performance drop when applying completion to all queries, compared to completing the occluded queries classified by the occlusion classification network. This is because the Completion Network is designed to reconstruct the occluded queries to learn occlusion-tolerant visual representations. Consequently, applying completion to non-occluded queries introduces confusion, degrading overall 3D detection performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SLIqNwuuOC", "reviewer": "Reviewer_g4DX", "summary": "This paper introduces a novel framework for improving monocular 3D object detection, particularly in handling object occlusions. The proposed MonoMAE leverages depth-aware masking to simulate occlusions in the feature space and employs a lightweight completion network to reconstruct occluded object regions, thereby learning occlusion-tolerant representations. Experiments show that this learning stratgy helps to improve the performance of monocular 3D object detection.", "questions": "1. All the experimental results presented in this paper are about vehicle detection. Does MonoMAE also work for more difficult cases like pedestrain and cyclist detection? \n2. The paper suggests investigating generative approaches for simulating natural occlusion patterns. Can you elaborate on what this might entail and how it could further improve monocular 3D detection?", "limitations": "1. The paper could provide a more detailed analysis of the computational efficiency, including speed and resource usage, to fully assess the practicality of MonoMAE for real-time applications.\n2. This paper only present results of vehicle detection. The performance of detecting objects with small sizes is unknown.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper is well-structured, with a clear problem statement, methodology, experiments, and ablation studies that substantiate the contributions and effectiveness of MonoMAE.\n2. This paper addresses a significant challenge in monocular 3D object detection, object occlusion, with a novel approach using depth-aware masked autoencoders.", "weaknesses": "1.  The reliance on depth-aware masking to simulate occlusions may not perfectly replicate natural occlusion patterns, potentially affecting the model's reconstruction accuracy. The gap between synthetically masked and naturally occluded object queries could limit the model's robustness in real-world scenarios.\n2. While this paper claims generalizability, the lack of extensive cross-dataset validation leaves the true scope of its generalization capability somewhat unproven.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.\n\n**Weakness 1: Depth-Aware Masking for Occlusion Simulation**\n\nWe acknowledge that the proposed depth-aware masking may not perfectly replicate natural occlusion patterns, which could affect the model's reconstruction accuracy potentially.\n\nTo fill the gap between synthetic and natural occlusions, we have considered alternative methods, such as directly overlaying images of objects (e.g., using a car image to occlude another car). However, this approach poses significant challenges, particularly in obtaining accurate training labels (3D boxes, orientations) for the overlaying objects, making it impractical for our current implementation. \n\nAs shown in Section D.1 of the submitted Appendix, one possible solution is introducing Generative Adversarial Networks (GANs) that learn distributions from extensive real-world data for generating occlusion patterns that are very similar to natural occlusions. Specifically, a Generator could be employed to generate occluded queries based on non-occluded queries, while a Discriminator could be used to discriminate between these generated occluded queries and natural occluded queries. Through adversarial training, the Generator could produce occlusions that are very similar to natural occlusions.\n\n\n**Weakness 2: Lacking Extensive Cross-Dataset Validation**\n\nWe would clarify that most existing monocular 3d detection methods [1, 2, 3] validate their generalization ability on the task KITTI3D->nuScenes only. We followed these prior studies to facilitate benchmarking. As suggested, we extend the experiments by examining the generalization over a new task KITTI3D->Waymo as shown in the table below, using $AP(IoU=0.5)$ as the metric. We can observe that our method has generalization ability on the Waymo dataset, and even outperforms some methods trained on Waymo.\n\n| Method | PatchNet* [4] | M3D-RPN* [5] | Ours  |\n|--------|------|----------|------|\n|  Level_1  | 2.92    |   3.79     |  **4.53**    |\n|  Level_2  | 2.42   |    3.61     |  **4.17**    |\n\n\\* denotes this method is trained on Waymo. \n\n[1] Shi, Xuepeng, et al. Geometry-based distance decomposition for monocular 3d object detection. ICCV, 2021.\n\n[2] Kumar, Abhinav, et al. Deviant: Depth equivariant network for monocular 3d object detection. ECCV, 2022.\n\n[3] Jinrang, Jia, et al. MonoUNI: A unified vehicle and infrastructure-side monocular 3d object detection network with sufficient depth clues. NeurIPS, 2023.\n\n[4] Ma, Xinzhu, et al. Rethinking pseudo-lidar representation. ECCV, 2020.\n\n[5] Brazil, Garrick, et al. M3d-rpn: Monocular 3d region proposal network for object detection. ICCV, 2019.\n\n**Question 1 & Limitation 2: Performance on Small Size Objects (Pedestrian and Cyclist)**\n\nThe proposed MonoMAE can handle challenging cases with competitive detection performance. We conducted new experiments on the suggested pedestrians and cyclists that often have smaller scales and are more challenging to detect. The two tables below show the experimental results with metric $AP_{3D}(IoU=0.7)$.\n\nPerformance for pedestrian detection.\n| Method | Easy | Moderate | Hard |\n|--------|------|----------|------|\n|  DID-M3D  |  11.78    |    7.44      |   6.08   |\n|    MonoNeRD    |  13.20    |   8.26       |  7.02    |\n| Ours       |   **13.37**   |    **8.41**      |    **7.10**  |\n\nPerformance for cyclist detection.\n| Method | Easy | Moderate | Hard |\n|--------|------|----------|------|\n|  DID-M3D  | 7.82     |    3.95      |  3.37    |\n|  MonoNeRD     | 4.79     |  2.48        |  2.16    |\n| Ours       |  **8.05**    |  **4.16**        |  **3.54**    |\n\n\nWe can observe from the above two tables that the proposed MonoMAE performs well for the pedestrian and cyclist categories for monocular 3D object detection.\n\n\n**Question 2: Generative Approaches for Simulating Natural Occlusion Patterns**\n\nAs briefly shared in Section D.1 of the submitted Appendix, the proposed MonoMAE could be improved by employing generative networks such as GANs to learn distributions of real-world data. The trained model will then generate occlusion patterns that are more similar to natural occlusions than our proposed feature masking, leading to better monocular 3D object detection. Take GAN as an example. The GAN generator will learn to generate occluded queries (with many non-occluded queries as reference), while the GAN discriminator will learn to discriminate between the generated occluded queries and naturally occluded queries. Through adversarial learning, the trained generator could generate more realistic occlusions, which further leads to better occlusion completion and monocular 3D object detection.\n\n\n**Limitation 1: Computational Efficiency**\n\n\nWe would clarify that we have provided the inference time of the proposed MonoMAE and several state-of-the-art monocular 3D detection methods in Table 5 of the submitted manuscript. According to the inference time, MonoMAE can achieve above 26.3 FPS (frame per second) which demonstrates its great potential for real-time tasks. For resource usage, all our experiments were conducted on a computer with Intel Xeon Gold 6134 CPUs, CentOS operating system with 128GB RAM, and a single NVIDIA V100 GPU with 32 GB memory. We will highlight the above information in the updated manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HjirQrSA9G", "reviewer": "Reviewer_bA8P", "summary": "This paper proposes a monocular 3D detection framework inspired by Masked Autoencoders (MAE), designed to address the challenge of object occlusions in 3D object detection. It utilizes a unique depth-aware masking module that simulates occlusions by adaptively masking non-occluded object features based on depth information, coupled with a lightweight completion network that reconstructs these masked features to learn occlusion-tolerant representations. It generates training pairs of non-occluded and occluded object representations directly, enhancing its capability to handle occlusions effectively. The framework is optimized for low computational overhead during inference, as it does not require object masking at this stage.", "questions": "1. Many recent 3D object detection studies have utilized the Waymo dataset for evaluation. Could you explain why your experiments were limited to KITTI and nuScenes?\n2. There appears to be a performance drop in the nuScenes dataset at distances beyond 40 meters. Could you provide insights into what causes this decline?\n3. There is a slight difference in inference time between 'Ours*' (36ms) and 'Ours' (38ms), with significant performance differences noted in Table 2. Could you elaborate on the role of the completion network (CN) given these differences?\n4. The mask ratio r varies with scale parameters and maximum depth. How sensitive is your method to changes in the mask ratio?", "limitations": "See the weakness and question parts.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method outperforms the conventional methods across various datasets such as KITTI and Nuscenes. It demonstrates the effectiveness of the proposed method. Moreover, the proposed method achieves real-time inference time.\n2. An extensive ablation study is proven to demonstrate the impact of the proposed module. \n3. The idea is simple yet effective.", "weaknesses": "1. The performance improvement is marginal, especially on the cross-validation in Table 6. \n2. Missing evaluation on the Waymo dataset", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.\n\n\n**Weakness 1: Performance Improvement**\n\nWe would clarify that the proposed MonoMAE achieves clear performance improvement over the state-of-the-art. As shown in Table 1 of the submitted manuscript, MonoMAE outperforms the SOTA method MonoCD [1] by clear margins, especially for objects of Moderate and Hard categories. The scale of the improvement is also competitive as compared with several recently published works as listed in Table 1.\n\n\nThe experiments in Table 6 aim to validate the generalization ability of the proposed MonoMAE. We can observe that MonoMAE achieves the best or the second-best performance consistently across all metrics. Besides, the MAE metric used in Table 6 is hard to obtain significant increases. For example, MonoUNI [2] obtains slight performance gains and even performance drops on the nuScenes dataset.\n\n[1] Yan, Longfei, et al. MonoCD: Monocular 3D Object Detection with Complementary Depths. CVPR, 2024.\n\n[2] Jinrang, Jia, et al. MonoUNI: A unified vehicle and infrastructure-side monocular 3d object detection network with sufficient depth clues. NeurIPS, 2023.\n\n\n**Weakness 2 & Question 1: Experiments on Waymo**\n\nThanks for your suggestion. We extend the experiments by examining the generalization over a new task KITTI3D->Waymo as shown in the table below, using $AP(IoU=0.5)$ as the metric. We can observe that our method has generalization ability on the Waymo dataset, and even outperforms some methods trained on Waymo.\n\n| Method | PatchNet* [1] | M3D-RPN* [2] | Ours  |\n|--------|------|----------|------|\n|  Level_1  | 2.92    |   3.79     |  **4.53**    |\n|  Level_2  | 2.42   |    3.61     |  **4.17**    |\n\n\\* denotes this method is trained on Waymo. \n\n[1] Ma, Xinzhu, et al. Rethinking pseudo-lidar representation. ECCV, 2020.\n\n[2] Brazil, Garrick, et al. M3d-rpn: Monocular 3d region proposal network for object detection. ICCV, 2019.\n\n**Question 2: Performance Drop at Far Distances on the nuScenes dataset**\n\nThe performance drop beyond 40 meters could be due to: \n1) Limited visual information (in image resolution, etc.) is captured for objects beyond 40 meters; \n2) Adverse conditions in lighting and bad weather could have a larger impact on the quality of distant objects; \n3) The annotation accuracy could degrade for distant objects due to small object sizes, leading to degraded network training.\n\nAs shown in Table 6 of the submitted manuscript, existing SOTA methods all have performance drop in the nuScenes dataset at distances beyond 40 meters. The proposed MonoMAE achieves the second-best performance at a far distance.\n\n\n\n**Question 3: The Role of the Completion Network**\n\nThe Completion Network aims to learn to complete occluded query objects for detection performance for various occluded objects. It is a lightweight network that introduces 2ms additional inference time only but can achieve effective completion in the feature space. During training, it learns to reconstruct and complete the masked object queries (simulating real object occlusions). During inference, it reconstructs the occluded queries as completed ones which clearly improves the performance of monocular 3D detection as shown in Table 2.\n\nWe managed to show the impact of the completion network by plotting the losses with and without using the completion network in Figure 7 of the submitted Appendix. The graph shows that the training loss with the Completion Network (the orange line) drops while the training loss without the Completion Network (the blue line) remains at a high level, showing that the Completion Network helps acquire occlusion-tolerant representations by learning to reconstruct the masked queries.\n\n\n\n**Question 4: Sensitivity of Our Method to the Mask Ratio**\n\nAs defined in Equation (3) of the submission, the mask ratio $r$ of each query is determined by\n$r = 1.0 - \\frac{d_{i}}{D_{max}}$. $r$ is the applied mask ratio for each query. $d_{i}$ is the predicted depth of the $i$-th query. $D_{max}$ is the maximum depth in the dataset that could manually adjusted to affect the mask ratio $r$.\n\n\nThe table below shows the performance of our method when changing the maximum depth $D_{max}$. In each cell of the table, the performance is listed as $AP_{3D}(IoU=0.7)$ / $AP_{BEV}(IoU=0.7)$.\n\n| Maximum Depth | Easy          | Moderate      | Hard          |\n|-----------------------------------|---------------|---------------|---------------|\n| $0.5*D_{max}$                     | 28.17 / 38.21 | 19.83 / 26.62 | 17.06 / 21.24 |\n| $0.75*D_{max}$                    | 29.63 / 39.78 | 20.31 / 26.85 | 17.42 / 22.70 |\n| $D_{max}$                         | **30.29** / **40.26** | **20.90** / **27.08** | **17.61** / **23.14** |\n| $1.5*D_{max}$                     | 29.04 / 39.23 | 19.92 / 26.43 | 17.15 / 22.39 |\n| $2.0*D_{max}$                     | 27.70 / 37.18 | 19.45 / 26.12 | 16.84 / 20.52 |\n\nWe can observe that the best performance is achieved when the maximum depth is set at the original $D_{max}$. When the maximum depth is smaller than $D_{max}$, the performance drops. This is because if the depth of an object is too large, the query of this object will not be masked, leading to masking failures and hindering the training of the completion network.\n\nWhen we set the maximum depth larger than $D_{max}$, the performance also drops. This is because the mask ratios are relatively high with the large maximum depth for all objects according to Equation (3). A high mask ratio will pose challenges for the Completion Network by reducing the available information since objects at far normally have limited pixels, hindering the reconstruction of occluded object regions. \n\nNevertheless, we can observe that setting the maximum depth between $0.75$ and $2$ $*D_{max}$ achieves similar 3D detection, indicating the robustness of our method with respect to this parameter.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We thank all the reviewers for your time, insightful suggestions, and valuable comments. We are highly encouraged by the reviewers' acknowledgment of our method in its innovative idea and novel design (xT7k, 8dEd, g4DX), superior performance (xT7k, 8dEd), exhaustive experiments (g4DX, bA8P), strong generalization capabilities (xT7k) and good presentation (8dEd, g4DX). Most reviewers concurred that the studied occlusion problem is critical and a significant challenge for monocular 3D detection.

The reviewers also shared several concerns, mainly focusing on:
- More detailed information on the usage of occlusion levels information.
- Further insights regarding the gap between synthetic and natural object occlusion.
- Additional experiments regarding the network generalization.

We respond to each reviewer's comments in detail below. We again thank the reviewers for your valuable suggestions, which we believe will greatly strengthen our paper.

Meta-review: The paper receives positive ratings from all the reviewers. They liked the idea of using depth-aware masked auto-encoders for monocular 3D object detection. The technical contribution is solid, and the experimental results are strong. AC agrees with the positive comments from the reviewers and recommends 'accept as poster' for this paper.
Decision: Accept (poster)