WeakM3D: Towards Weakly Supervised Monocular 3D Object Detection
Authors: Liang Peng
Senbo Yan
Boxi Wu
Zheng Yang
Xiaofei He
Deng Cai
Venue: ICLR 2022 Poster (2022)
Keywords: Computer vision
monocular 3D object detection
weakly supervised

Abstract: 	Monocular 3D object detection is one of the most challenging tasks in 3D scene understanding. Due to the ill-posed nature of monocular imagery, existing monocular 3D detection methods highly rely on training with the manually annotated 3D box labels on the LiDAR point clouds. This annotation process is very laborious and expensive. To dispense with the reliance on 3D box labels, in this paper we explore the weakly supervised monocular 3D detection. Specifically, we first detect 2D boxes on the image. Then, we adopt the generated 2D boxes to select corresponding RoI LiDAR points as the weak supervision. Eventually, we adopt a network to predict 3D boxes which can tightly align with associated RoI LiDAR points. This network is learned by minimizing our newly-proposed 3D alignment loss between the 3D box estimates and the corresponding RoI LiDAR points. We will illustrate the potential challenges of the above learning problem and resolve these challenges by introducing several effective designs into our method. Codes are available at https://github.com/SPengLiang/WeakM3D.


Published as a conference paper at ICLR 2022
WEAKM3D: TOWARDS WEAKLY MONOCULAR 3D OBJECT DETECTION
SUPERVISED
Liang Peng Senbo Yan 1. Boxi Wu', Zheng Yang,, Xiaofei He & Deng Cai State Key Lab of CAD&CG, Zhejiang University
Senbo
FABB Inc.
(pengliang, yansenbo, wuboxi jz edu. en ef ai xiiaofeihe, dengcai @ead, 2 ju. edu. cn
ABSTRACT
Monocular 3D object detection is one of the most challenging tasks in 3D scene understanding. Due to the ill-posed nature of monocular imagery, existing monoc- ular 3D detection methods highly rely on training with the manually annotated 3D box labels on the LiDAR point clouds. This annotation process is very laborious and expensive. To dispense with the reliance on 3D box labels, in this paper we explore the weakly supervised monocular 3D detection. Specifically, we first de- tect 2D boxes on the image. Then, we adopt the generated 2D boxes to select corresponding Rol LiDAR points as the weak supervision. Eventually, we adopt network to predict 3D boxes which can tightly align with associated Rol LiDAR points. This network is learned by minimizing our newly-proposed 3D alignment loss between the 3D box estimates and the corresponding Rol LiDAR points. We will illustrate the potential challenges of the above learning problem and resolve these challenges by introducing several effective designs into our method. Codes are available at https//gtttbbboomm//P/ggggagaaaa
INTRODUCTION
3D object detection is essential for many applications in the real world, such as robot navigation and autonomous driving. This task aims to detect objects 3D space, bounding them with oriented 3D
boxes. Thanks to low deployment cost, monocular-bese based methods Chen et al. (2016)][Musay))) Mousavvan
ct al. (017)): [Roddick et al. (2018); Liu et al.] (2019); Manhardt et al. (2019): Ma et al.) (2020) 2021 are drawing increasing attention in both academia and industry. In recent years, monocular
3D detection has achieved remarkable progress. However, it also requires numerous 3D box labels (locations, dimensions, and orientations of objects) for training. These labels are labeled LiDAR point clouds, where the manually annotating process is quite time-consuming and expensive. The high annotation cost encourages us dispense with the reliance the 3D box annotations.
To this end, in this paper we propose WcakM3D, a novel method towards weakly supervised monoc- ular 3D object detection. Considering the well-developed 2D detection technology Redmon & Farhadi (2018): Ren et al. )0133) He et al. (017); Qi et al. (2018); Zhou et al. (2019). we use an off-the-shelf 2D detector Qi et al. (2018) obtain 2D boxes, which are then lifted to 3D boxes by predicting required 3D box parameters. To learn the 3D information desired in the lifting process, we employ the LiDAR point cloud as the weak supervision since provides rich and accurate 3D points within the scene. Specifically, given a raw LiDAR point cloud, we select LiDAR points if their projections are inside the 2D object on the image plane. We term these points object-LiDAR- points, which describe part outline of the object in 3D space. Therefore, as shown in Figure (a), object- object-LiDAR-aa points can be used for aligning with 3D box predictions in loss functions, conse- quently endowing the network with the ability of being trained without any 3D box label.
However, how to formulate the loss between 3D box predictions and Biteec-LiDAR-poine is con- siderably challenging. We pose and summarize the main challenges and our solutions.
Challenge 1 (Section 3.2): as shown in Figure I (a), the alignment gap between 3D box estimates and Lbject-L.iDAR-poiine should be measured appropriately. We address this challenge by introduc-

---

Published as a conference paper at ICLR 2022
Aligring
Oeject-LIARR-ooinn
Sparse yet important points
Aligning with whish surfnce?
Surface
Objact-LIDARR pollnts
Dense points
box prediction (ofter training)
Paint-to-box loss
3D box prodiction
3D box prediction
Objec--iDARRppoingg
(c)
Figure 1: We use the LiDAR point cloud as the weak supervision in training. (a): the aligning process; (b): the alignment ambiguity problem; (c): unevenly distributed LiDAR points.
ing a geometric point-to-hox alignment loss, to minimize spatial geometric distances from 3D box predictions to Doject-LiDAR-poinn This loss allows the network to learn the object's ss 3D location.
Challenge (Section 3.3) as shown in Figure (b), an alignment ambiguity problem caused if only geometrically aligning 3D box predictions with Dbject-LiDAR-oints. Regarding object- LiDAR-points that are captured from only one surface of an object, is not clear which surface of the 3D box prediction should be used for aligning. thus causing alignment ambiguity issue. Inspired by the constraints of camera imaging and LiDAR scanning, we eliminate this ambiguity track each ahject-LiDDAR
by proposing ray tracing loss. In particular, optical center and use the resulting ray to make collision detection with predicted 3D boxes to find the surface correspondence.
from the camera
this way, the alignment ambiguity issue can be alleviated greatly.
Challenge 3 (Section 3.4): as shown in Figure (c). LiDAR point clouds distribute in 3D space unevenly. Considering that both geometric alignment and ray tracing losses calculated point- wisely, unevenly distributed points can cause unbalanced point-wise losses in training, which is harmful since losses produced by sparse yet important points are overwhelmed by losses of dense points. We use the point density to balance point-wise losses to resolve this problem.
Challenge (Section 3.5): a 3D box is parameterized by many estimates (locations, dimensions, and orientations). Such entangled 3D box estimates result in a heavy learning burden during training. To resolve this issue in training, we disentangle the learning of each group of estimates by freezing the object dimension and heuristically obtaining the orientation from object-LiDAR-poniee -points.
Such challenges and our method are detailed in the following sections. Extensive experiments vali- date the effectiveness of our method. In summary, our contributions can be listed as follows: Firstly, we explore a novel method (WeakM3D) towards weakly supervised monocular 3D detection, remov- ing the reliance on 3D box labels. Secondly, we pose the main challenges in WeakM3D and corre- spondingly introduce four effective strategies to resolve them, including geometric alignment loss, ray tracing loss, loss balancing, and learning disentanglement. Thirdly, evaluated the KITTI benchmark, our method builds a strong baseline for weakly supervised monocular 3D detection, which even outperforms some existing fully supervised methods which use massive 3D box labels.
RELATED WORK
2.1 LIDAR-BASED 3D OBJECT DETECTION
The LiDAR device is able to provide point clouds with precise depth measurements for the scene. Thus, LiDAR-basse methods Shi et al.] (2019b); [ang et al.] (2019): He et al. (2020); Shi et al. (020): Shi & Rajkumar (2020): Shi et al. (2019a); Zheng et al. (2021) attain high accuracy and can employed in autonomous driving. Early methods project point clouds into the bird's-eye- view Chen et al. (2017b) or front-view Li et al. (2016). ignoring the nature of point clouds, thus resulting in sub-optimal performances. LiDAR-based 3D detectors can be roughly divided into two categories; voxel-based methods Zhou & Tuzel (2018); : Yan et al. (2018); Kuang et al.) (2020) and point-based methods Shi et al. (2019)): Qi et al. (2018); Yang et al. 2019). The former partition the 3D space into voxel grids, transforming the irregular raw point cloud to regular voxels that 3D convolutions can be employed to extract more discriminative features. Point-based methods directly design the network tailored to the point cloud representation. Both two types of methods have
2

---

Published as a conference paper at ICLR 2022
achieved great success, but inherently limited by the main shortcomings of the LiDAR device, i.e, the high price and limited working ranges.
2.2 MONOCULAR-BASED 3D OBJECT DETECTION
In recent years, monocular 3D object detection has achieved significant improvements Qin et al. (2019); [Bazzil & Liu (2019); Ma et al. (202T). Prior works such as Mono3D Chen et al. (2016) and Deep3DBox Mousavian et al. (2017) mainly take advantage of geometry constraints and aux- iliary information. More recently, Monodle Ma et al. (2021) resorts to reducing the localization error in monocular 3D detection by three tailored strategies. Furthermore, with the development of depth estimation, some other monocular methods attempt to the explicit depth information generated from an off-the-shelf depth estimator. Pseudo- -iDuRo-LiDAR Wang et al. (2019) 9); Weng & Kitani (2019) converts the image-only representation, to mimic the real LiDAR signal to utilize the existing LiDAR-based 3D detector. PatchNet Ma et al. (2020) rethinks the underlying mechanism of pseudo LiDAR, pointing out the effectiveness comes from the 3D coordinate transform. Although recent monocular 3D object detection methods obtain exciting results, they heavily rely on a large number of manually labeled 3D boxes.
2.3 WEAKLY SUPERVISED 3D OBJECT DETECTION
To alleviate heavy annotation costs, some weakly supervised methods are proposed. WS3D Meng it al. (2020) introduces a weakly supervised approach for LiDAR R-based 3D object detection, which still requires small set of weakly annotated scenes and few precisely labeled object instances. They use a two-stage architecture, where stage one learns to generate cylindrical object proposals under horizontal centers
in bird's sird'--eye-view and stage two learns to refine the cylin- get 3D boxes and confidence scores. This weakly supervised design does not
drical proposals fully get rid of the dependence on 3D box labels and only works well for LiDAR point clouds input. Another weakly supervised 3D detection method Qin al. (2020) also takes point clouds as input. proposes a 3D proposal module and utilizes an off-the-sheif 2D classified network to identify 3D box proposals generated from point clouds. Both two methods cannot be directly applied fit the single RGB image input. Also, Zakharov et al. Zakharov et al. (2020) propose an autolabel- ing pipeline. Specifically, they apply novel differentiable shape renderer signed distance fields (SDF), leveraged together with normalized object coordinate spaces (NOCS). Their autolabeling pipeline consists of six steps, with curriculum learning strategy, In contrast WcakM3D, their method is not end-to-end and rather complicated.
3 METHODS
3.1 PROBLEM DEFINITION
Given an RGB image captured from a single calibrated camera and the corresponding projection matrix P, the task, ie., monocular 3D object detection, aims to classify and localize objects within the scene in 3D space, where each object is represented by its category cls, 2D bounding box b2d, and 3D bounding box bad. In particular, we utilize a separate 2D detector achieve 2D bounding boxes of objects, eliminating the extra attention on this well-developed area. This simple design enables us to focus on solving the 3D bounding box, which is the most important and challenging in monocular 3D detection. Note that we do not use any 3D box label during training. Specifically, the 3D bounding box bas is parameterized by the location (Izd, 3/d, 23d), the dimension (had, W3d laa), and the orientation (0,), which are all described at the camera coordinate system in 3D space.
We show our network architecture in Figure 2 and the forward pass is as follows: deep features are firstly extracted from single RGB image via encoder. At the same time, the image is fed into an off-the-shelf 2D detector Qi et al. (2018) to generate 2D bounding boxes. The 2D boxes are then used to obtain specified object features for every object from deep features. Finally, we achieve the 3D box bad by regressing the required 3D box parameters from object features.
Object-LiDAR-point contain only LiDAR points located on the object sur- face, indicating much 3D information. Thus they are ideal weakly supervised signals for monocular 3D detection. We achieve Lbject-LiDDAR-ponns from raw LiDAR point clouds and an off- the--sh--lf
Loject-LiDAR-poine
3

---

Published as a conference paper at ICLR 2022
3D Feature Encoder
Deep Features
RGB Image
Object Features
3D Box Predictions
Rol Align
Boxes
2D Detector
Geometric alignment loss
Ray tracing loss
Point-wise loss balancing
Learning disentanglement
Weak Supervision
Averaged Loss
Single Object
LIDAR Point Cloud
ObjecHLiDAR-Poing
Network architecture. In the inference stage, we require only single RGB image as input and output corresponding 3D boxes. In the training stage, we use LiDAR point cloud and 2D boxes estimated from a pre-treinne pre-trained model obtain Abject-iDDAR-poinss. which are used build losses with 3D box predictions to train the network. Best viewed in color.
Figure 2:
2D detector. First, the ground plane is estimated from the raw LiDAR point cloud via RANSAC chler & Bolles (1981), which used to remove ground points in the raw point cloud. The remaining LiDAR points whose projections on the image within the 2D box are selected as the initial object point cloud, which contains some points of no interest such as background points. Alternatively, we also 2D instance masks from pre-trained network to select the initial object point cloud. Then, we obtain object-LiDAR -oints by filtering the initial point cloud via an unsupervised density clustering algorithm Ester et al. (199)), which is detailed in the appendix.
3.2 GEOMETRICALLY ALIGNED 3D Box PREDICTIONS
Without loss of generality, 3D object boxes should contain object- LiDAR-points and align with them along box edges. To facilitate the learning of the object's 3D location, straint between 3D box predictions from the network and object- LiDAR-points. A naive solution refers to minimizing the euclidean distance from the object center to each LiDAR point. Nevertheless, this trivial loss misleads the network due its inaccurate nature for localization. As shown in Figure 3 this center distance loss Coenter would push the predicted object center the point cloud as close as possible. Unfortunately. object-LiDAR -ointts are typically captured from visible surfaces of objects, meaning that the predicted center using only Leenter tends to be close the groundtruth box edge, but center.
GT Center
impose the location
With treining pone on
Center
Center
Leater
Genter
Initial Box Prediction
Final Box Prediction (after training )
the groundtruth
Figure 3: Adverse impacts if using only the center loss. Blue points refer to object-LiDAR- points. Even given an ideal ini- tial 3D box that is close to the groundtruth, the network con- ducts much worse prediction after training. Best viewed in color with zoom in.
Based on the above analysis, we propose geometric alignment loss LiLAL--i--oinnn
between the predicted 3D box and associated in which the major obstacle lies in appropriately measuring the dis- tance from points to the 3D box. Aiming this, we create ray from the 3D box center Psd to each Lbject-LiDAR- p-pnn P. where the ray intersects with the edge of the box prediction at Pr. Therefore, the geometric alignment loss at each point among object-LiDARRpoiin is as follows:
Lycometry = =|pp Prll = || Intersect Rayrs, -o p, baa)ll
4

---

Published as a conference paper at ICLR 2022
Geo. Alignment Loss
Ray Tracing Loss
Loss Balancing
Prediction
-
BEV Box Prediction
(with 2 Bufferin)
Denaity
Object
Object LIDAR Points
(with Butfering
Center
Canter
Densitynt
Aligmment
Denaltye?
GEV Box Prediction
Loskgennt-ale
Center Logs
Trasing Loss
Lostaatoncing
Densitypotsr
Camara
and ray tracing loss (Section 3.3 with loss balancing (Section (.4) for each object-LiDAR-pont and the corresponding 3D box prediction. The geometric alignment loss focuses on tightly aligning 3D box predictions with object-LiDAR-poing ing/LiDDAR scanning into consideration, to eliminate alignment ambiguity (See Figure 5 in the scene. Best viewed in color with zoom in.
Figure 4: Loss design. We introduce geometric alignment loss (Section 3.2)
The ray tracing loss further takes occlusion constraints of camera imag-
where Intersecl (Rayess- bsa) refers to the intersection point Pj. In particular, we do not di- rectly predict the absolute object's 3D center Pad, but its projection [tz, tyl on the image plane and the object instance depth Thus, Pad (tt ((t--c) (1v-cy
fx 2, 2)T, where fas fy, Cxs Cy the
fe
parameters from the camera projection matrix P.
We illustrate the geometry alignment loss in Figure with object-LiDAR points terms of geometry.
This loss forces 3D box predictions to align
3.3 RAY TRACING ALIGNMENT AMBIGUITY ELIMINATING
Although the geometric alignment bounds 3D box predic- tions and object-LiDAR-points The ambiguity happens when object-LiDAR-poins fail to represent an adequate 3D outline of the object, e.g., only one surface an object. Specifically, the ambiguity refers to how to semantically decide the correspondence be- tween each bbject-LiDDRRpoinn and edges of the box predic- tion in the aligning process. We call this problem the align- ment ambiguity.
captured from
Edgat
Edgel
BEV Box
Centert
OojectsLiDARppoint
Erigo
Figure 5 shows an example. Both the BEV box and com- mendably align with object-LiDAR R-iintss by the edge 4 and respectively. They conduct the same geometric alignment losses, while their 3D locations are quite different. However, we cannot decide which box better prediction only con- cerning the geometric alignment because existing geometric clues cannot indicate the semantic correspondence of object- LiDAR-porte points. In other words, there is dilemma in choosing the correspondence between box edges and object-LiDAN points. This ambiguity brings adverse impacts in training, especially for hard samples (their associated object-LiDAR- points usually carry poor information).
Aligning with which odge?
Figure 5: Alignment ambiguity when aligning with Lbject-LiDARRpointsss Both box 0 and produce the same geometric alignment losses. Best viewed in color with zoom in.
5

---

Published as a conference paper at ICLR 2022
Generally, resolve the alignment ambiguity by considering occlusion constraints. Similar to the process of camera imaging, when scanning a scene, the signal of a LiDAR device will be reflected meeting obstacles. Considering the reflected LiDAR signal inside the camera FOV (field of view), we propose to implement ray tracing from the camera optical center Pram to each object-LiDAR- point, minimizing the distance from each object-LiDDRRpon box. As shown in Figure 4 the LiDAR point is P' and the intersection point is PR (with Z buffering, namely, the closer intersection point is chosen), the ray tracing loss as follows:
to the intersection point the object
||P- PRIl1
if Raypeam P intersects with bsd.
Cray-tracing
otherwise.
(Pi, P2] Intersect( Rayre PP, bzd), and Pn P if Pi is closer to the camera, or P3 other- wise. Note that the ray tracing loss is zero if the ray does not intersect with the predicted 3D box, meaning that this loss here does not contribute to the gradient descent in pack-propagation. way. eliminate the alignment ambiguity, encouraging 3D box predictions to follow occlusion constraints in parallel to geometrically aligning with object-LiDAR points. Let us back to the ex- ample shown in Figure | The ray tracing losses produced by box 1 are much larger than losses of box O, consequently leading the network to conduct a reasonable and precise result.
this
3.4 POINT-WISE LOSS BALANCING
The varied spatial distribution of object-LiDAR LiDAAR-piitts is also an obstacle, i.e., the point density is somewhere high while somewhere low. Point-wise losses such the geometric alignment loss and ray tracing loss all suffer from the unevenly distributed nature. Specifically, the loss produced by the dense region can dominate the total loss, ignoring the loss conducted by other relatively sparse yet essential points. To balance the influence, we normalize the density of object-LiDAR -pintt when calculating loss. Let calculate the number of LiDAR points & in the neighborhood at point P; as follows:
& Σ I P3112 < R)
where M is the number of object-LiDARRponnts From Equation 3 the spatial range R towards the point P; are counted for the density. Thus we balance point-wise losses by weighting each point with the density, as follows:
we can know that points within
M Σ M
geammetyy + traay-tracing; + X
Lbatancing
(4)
&
We illustrate an example in Figure The balanced loss alleviates the adverse impact brought by the unevenly distributed LiDAR point clouds. Here we employ the center loss for regularization and empirically set to O. and set the coefficients of geometric alignment and ray tracing loss to by default.
3.5 LEARNING DISENTANGLEMENT FOR 3D Box ESTIMATES
As mentioned in Simonelli et al. 2019), complicated interactions of 3D box estimates take a heavy learning burden during training. Prior fully supervised methods can easily deal with this issue by disentangling the learning of each group of 3D box estimates, iee. imposing losses between each group of predictions and corresponding labels, respectively. However, it is not trivial for the weakly supervised manner due to the absence of 3D box labels. To disentangle the learning, we extract object orientations from LiDAR point clouds. In addition, we use a-priori knowledge about dimen- sions the specified category and freeze them because objects that belong to the same class have close sizes, e.g., cars. Such information allows us to solve each group of estimates independently, largely reducing the learning burden.
Orientation Estimated from Object-LiDAR-poine We propose a simple yet effective method to obtain the global object orientation from object-LiDAR-ooine Intuitively, Lbject-LiDAR-poinn describe a part 3D outline of the object, implicitly indicating the object orientation 8,. We obtain the object orientation from the directions of paired points. Specifically, shown in Figure 6

---

Published as a conference paper at ICLR 2022
Orantalis-
Object poRpti
Orientatonn
Figure 6: Orientation distribution. We calculate the direction of each pair of points among object- LiDAR-points and draw the histogram. We can observe that the direction that is highest in the histogram is highly related to the object orientation. Best viewed in color with zoom
calculate the direction of each pair of Mbjjet-LiDDAR-oimnee drawing a histogram with respect to the direction range from 0 to The direction of paired points varies near the object orientation, and the direction a that is highest in histogram refers to 0y or 0g + T/2. The object orientation is further decided by the spatial offset dz of object-LiDAR-poine indicates the orientation close to 2 while a high offset denotes the orientation close to 0. We set the offset threshold to 3.0 meters by default. Therefore, we heuristically obtain the global object orientation, which can be used as the supervised signal for orientation estimates and employed in the process of 3D box alignment. More details can be found in the appendix.
along æ axis in 3D space. A low offset
3.6 NETWORK TRAINING
We conduct point- wise losses the bird s-ee-vviv (BEV) to simplify the task. We train the network to learn the object's location (æza, #3d) by Loalaneings which the most challenging and crucial. To further obtain the 3D location, along axis, denoted by VL. building the loss: Liocy SmoothI,(0 (9. VL), where ý is the network prediction. Thus, we obtain the overall loss formulation as:
averaging coordinates among object-LiDARR-
L = Liatancing Liocy + Corient
We empirically set coefficients for each loss item to by default. For the orientation, only the local orientation, i.e., the observation angle, can be directly estimated from the RGB image. Thus the network conducts local orientation estimates and then converts to global orientations using the predicted 3D center (I3d, 23d). The orientation loss Lorsent follows Deep3DBox Mousavian et al. 2017). We provide more details in the appendix.
4
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
Our method is implemented by PoTorc Paszke et al. (2019) and trained on a Titan V GPU, We use the Adam optimizer Kingma & Ba 2014) with initial learning rate of 10-4, We train our network for 50 epochs. To obtain an initial object point cloud, we adopt Mask-RCNN He et al. (0177) pre- off-the-shelf 2D detector Qi et al.
trained on COCO Lin et al. (2014). Alternatively,
2018). Both manners lead comparable and satisfactory performance. For the frozen dimensions for cars, empirically adopt 1.6. 1.8, 4.0 meters the height, width, and length, respectively. R for the point density in Equation Bs set to 0.4-meter. We adjust coordinate in the location according 2D-3D consistency Brazil & Liu (2019). We provide more experimental results and discussion in the appendix due to the space limitation.
4.2 DATASET AND METRICS
Like most prior fully supervised works do, we conduct experiments on KITTI Geiger et al. (2012) dataset. KITTI object dataset provides 7,481 images for training and 7,518 images for testing, where groundtruths for testing are kept secret and inaccessible. Following the common practice Chen et al. (2017a). the 7,481 samples are further divided into training and validation splits, containing 3,7 and 3,769 images, respectively. To utilize raw LiDAR point clouds since our method does not require 3DD box labels, we use sequences for training that does not overlap with the validation and test

---

Published as a conference paper at ICLR 2022
Table 1: The experimental results KITTI validation for category. We observe that our method even outperforms some fully supervised methods. All the methods are evaluated with metric APIRn
Ru as many prior fully supervised works only provided AP R11 results.
AP AP (IoU=0.7) Ru1
Approaches
Supervision
Easy
Moderate
Hard
Mono3D Chen et al.
(0166)
5,22/2.53
5,19/2.31
4,13/2.31
Doep3DBox Mousavian
2017)
9.99/5.85
7.71/4.10
5.30/3.84
OFTNet Roddick (2018)
11.06/4.07
8.79/3.27
8.91/3.29
Rol-10D Manhardi
201
14.50/10.25
9.91/6.39
8.73/6.18
MS Simonelli al.
2019
24.26/18.05
18,43/14,98
16.95/13.42
FQNet Liu et al. (2019)
9.50/5.98
8.02/5.50
7.71/4.75
MonoPSR Ku (2019
20.63/12.75
18.67/11.48
14.45/8.59
Full
M3D-RPN Brazil Liu 2019
25.94/20.27
21.18/17.06
17.90/15.21
Pseudo-Lidar Wang el al.
2019
31.88/24.12
20.84/15.74
18.92/14,96
D4LCN Ding et al. (2020
26.00/19.38
20.73/16.00
17.46/12.94
RTM3D Li al. (2020)
25.56/20.77
22.12/16.86
20.91/16.63
PatchNet Ma
20200
32.30/25.76
21.25/17.72
19 19.04/15.62
Monodle Ma ct al.
2021
30.77/23.29
24.53/20.55
23.32/17.90
WcakM3D
24.89/17.06 29.89/20.51
16,47/11.63
14.09/11.17
Weak
SarkM3DDPPttchNe
18.62/13.67
16.06/12.02
Table 2: Comparisons on KITTI testing set for car category.
APBev/APs> (lo==077)
Approaches
Supervision
Easy
Moderate
Hard
FQNet et al. (2019)
5.40/2.77
3,23/1.51
2.46/1.01
Full
ROI- K01-10D 101 Manhardt et al. 2019)
9.78/4.32
4.91/2.02
3.74/1.46
W3D
Weak
11.82/5.03
5.66/2.26
4.08/1.63
set. We do not perform data augmentation during the training. Also, prior fully supervised mainly evaluate their methods on the category of car, and we follow this line. For evaluation metrics, APn commonly applied in many prior works, while AP4o Simonelli et al. (2019) is suggested to use recently. Most fully supervised methods use IOU 0.7 and weakly supervised methods use IOU 0.5 criterion, thus we follow them for comparisons, respectively.
4.3 COMPARING WITH OTHER APPROACHESS
Comparing with monocular- based fully supervised methods. As shown Table [ although our method does not any 3D box annotation, it outperforms some prior fully supervised methods.
For instance. WcakM3D outperforms
FONet Liu al. (019) by 8.45 APBEV
Table 3: Comparisons CenterNet for category.
under the moderate setting. Table [ also shows that our method outperforms some prior SOTA fully supervised methods. We can also observe that there still remains considerable gap between WeakM3D and current SOTA fully supervised methods Ma et al. (202)), we have faith in greatly pushing performance of weakly supervised methods in the future due the rapid development in the community. Furthermore, and CenterNet Zhou et al. (2019) in Table 3 while keeping other settings the same. Interestingly, weakly supervised method achieves comparable performance compared to the fully supervised manner, demonstrating its effectiveness.
We apply our method to CenterNet for comparisons.
APzev/APap (loU=0.7)) [
Supervision
Easy
Moderate
Hard
Full Weak (Ours)
3.47/0.60
3.31/0.66
3.21/0.77
5.27/1.23
3.99/0.79
3.36/0.71
apply method to PatchNet Ma et al. (2020) in Table [
Comparing with other weakly supervised methods. We compare method with other weakly supervised methods. We report the results in Table 4 To make fair comparisons, we employ our
baseline network for Autolabels Zakharov al. 2020 Note that the original VS3D Qin et al.
8

---

Published as a conference paper at ICLR 2022
Table 4: Comparisons of different weakly supervised methods on KITTI validation set for car category.
AP APsev/APsp (IoU=0.5) Rao
Approaches
Input
Supervision
Easy
Moderate
Hard
VS3D Qin et al. 2020)
Point cloud
31.59/22.62
20,39/14443
16.28/10.91
Autolabels Zakharov al. 2020)
Single image
Weak
50.51/38.31
30.97/19. 90
23.72/14.83
Ours
Single image
58.20/50.16
38.02/29.94
30.17/23.11
Table 5: Ablation study. "Disten." in the table refers the learning disentanglement, and "Geo. alignment" is the geometric alignment.
APaey/APsu (loU=0.5) 840
Disten.
Geo. alignment
Ray tracing
Loss balancing
Easy
Moderate
Hard
0.1 1/000
0.16/0.10
0.15/0.04
16.16/11.55 1.55
11.98/8.16 16
9.21/6.04 0.4
51.36/45.04
33.08/26.34
26.42/21.19
55.31/48.35
36.28/28.48
29.03/22.18
V
58.20/50.16
38.02/29.94
30.17/23.11
2020) takes advantage of the per-trained network Dorn Fu al. (2018) to estimate image-based
depth maps. As mentioned in Simonelli et al. (020): Peng et al. (2021), Dorn utilizes the leaked
data on the KITTI 3D object dataset. Therefore the number of LiDAR point clouds the same as our method for fairness. We can observe that our method performs the best, validating its effectiveness.
use the fixed split Peng et al. (202I) and keep
4.4 ABLATION STUDY
To investigate the impact of each component in our method, we conduct extensive ablation stud- ies. As shown in Table [S] the accuracy of the baseline is very low, indicating that the weakly supervised manner is challeng-
Table 6: Comparisons of different weak supervisions in training.
APREv/APsD (loU=0.5) Rao
Weak supervisions
Easy
Moderate
Hard
2D box LiDAR
55.94/47.52
34.11/26.46
25.56/19.88
2D mask + LiDAR
58.20/50.16
38.02/29.94
30.17/23.11
ing. By adding the learning disentanglement and geometry alignment between 3D box predictions and Litectt--iAR--oiii
we obtain significant progress. When employing ray tracing, which con- siderably eliminates the alignment ambiguity, the performance is further improved. Finally, the balancing at point-wise losses in favor of accuracy and stability makes the network perform the best. These results validate the power of our method. We also compare the weakly supervised manner ing different requirements in Table 6
We can see that both requirements lead to good performance.
5 LIMITATIONS AND FUTURE WORK
There are also some limitations in our work. The loss for 3D localization is a little complicated, and the accuracy be further improved by more elegant design. As emerging area, there is ample room for improving the accuracy of weakly supervised monocular 3D object detection. Therefore, we resort alleviating the label reliance further and boosting the performance future work.
CONCLUSIONS
In this paper, we explore novel weakly supervised monocular 3D object detection method. Specif- ically, we take advantage of 2D boxes/masks to segment LiDAR point clouds to obtain object- LiDAR-points, which are employed as the weakly supervised signal for monocular 3D object detec- tion. To learn the object's 3D location, we impose alignment constraints between such points and 3D box predictions. Aiming this, we point out the main challenges of the weakly supervised manner, introducing several novel strategies to deal with them. Extensive experiments show that our method builds a strong baseline for weakly supervised monocular 3D detection, which even outperforms some prior fully supervised methods.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported in part by The National Key Research and Development Program of China (Grant Nos: 2018AAA0101400), in part by The National Nature Science Foundation of China (Grant Nos: 62036009, U1909203, 61936006, 62133013), in part by Innovation Capability Sup- port Program of Shaanxi (Program No. 2021TD-05).
REFERENCES
Garrikk Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9287- 9296, 2019.
Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monoc- ular 3d object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2147-2156, 2016.
Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Huimin Ma, Sanja Fidler, and Raquel Urtasun. 3d object proposals using stereo imagery for accurate object class detection. IEEE transactions on pattern 2017a
analysis and machine intelligence,
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 1907-1915, 2017b.
Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang. Jianping Shi, Zhiwu Lu, and Ping Luo. Learning septh--uided convolutions for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11672-11681, 2020.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density- based algorithm for discovering clusters in large spatial databases with noise. In Kdd, volume 96, pp. 226-231, 1996.
Martin Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381 1981.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2002-2011, 2018
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 3354 3361. IEEE, 2012.
Chenhang He, Hui Zeng, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang. Structure aware single- stage 3d object detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11873-11882. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He,, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-emn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Jason Ku, Alex D Pon. and Steven L Waslander. Monocular 3d object detection leveraging accurate proposals and shape reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11867-11866, 2019.
10

---

Published as a conference paper at ICLR 2022
Hongwu Kuang, Bei Wang, Jianping An, Ming Zhang, and Zehan Zhang. Voxel-fpn: Multi-scale voxel feature aggregation for 3d object detection from lidar point clouds. Sensors, 20(3):704, 2020.
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpil- lars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12697-12705, 2019.
Bo Li, Tianlei Zhang, and Tian Xia. Vehicle detection from 3d lidar using fully convolutional network. arXiv preprint 1rXiv:1608.07916, 2016.
Peixuan Li, Huaici Zhao, Pengfei Liu, and Feidao Cao. Rtm3d: Real-time monocular 3d detection from object keypoints for autonomous driving. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part III 16, pp. 644-660 Springer, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Lijie Liu, Jiwen Lu, Chunjing Xu, Qi Tian, and Jie Zhou. Deep fitting degree scoring network for monocular 3d object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1057-1066, 2019.
Xinzhu Ma. Shinan Liu, Zhiyi Xia, Hongwen Zhang, Xingyu Zeng, and Wanli Ouyang. Rethinking pseudo-lidar representation. In European Conference on Computer Vision, pp. 311-327. Springer, 2020.
Xinzhu Ma. Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojic Li, and Wanli Ouyang. Delv- ing into localization errors for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4721-47300 2021.
Fabian Manhardt, Wadim Kehl, and Adrien Gaidon. Roi-10d: Monocular lifting of 2d detection to 6d pose and metric shape. In Proceedings of the IEEE Conference on Computer Vision and 2019.
Pattern Recognition, pp.
Qinghao Meng, Wenguan Wang, Tianfei Zhou, Jianbing Shen, Luc Van Gool, and Dengxin Dai. Weakly supervised 3d object detection from lidar point cloud. In European Conference Com- puler Vision, pp. 515-531. Springer, 2020.
Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box esti- mation using deep learning and geometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7074-7082, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in neural information processing systems, pp- 8026-8037, 2019.
Liang Peng, Fei Liu, Senbo Yan, Xiaofei He, and Deng Cai. Ocm3d: Object-centric monocular 3d object detection. arXiv preprint 4rXiv:2104.06041, 2021.
Charles Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 918-927, 2018
Zengyi Qin, Jinglu Wang, and Yan Lu. Monogrett: A geometric reasoning network for monoc- ular 3d object localization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8851-8858 2019.
Zengyi Qin, Jinglu Wang, and Yan Lu. Weakly supervised 3d object detection from point clouds. In Proceedings of the 28th ACM International Conference Multimedia, pp. 4144 4152-415,, 2020.
11

---

Published as a conference paper at ICLR 2022
Joseph Redmon and Ali Farhadi. arXiv:1804.0277, 2018.
Yolov3: An incremental improvement.
arXiv preprint
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. arXiv preprint arXi:: 2015.
Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic feature transform for monocular 3d object detection. arXiv preprint 1 1.08/88, 188, 2018.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-779, 2019a.
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. arXiv preprint arXiv:1907.03674 2019b.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng
Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10529-10538, 2020.
Weijing Shi and Raj Rajkumar. Point-gnn: Graph neural network for 3d object detection in point cloud. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1711-1719, 2020.
Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Manuel López-Antequera, and Peter Kontschieder. Disentangling monocular 3d object detection. In Proceedings of the IEEE In- lernational Conference on Computer Vision, pp. 1991-1999, 2019.
Andrea Simonelli, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, and Elisa Ricci. Demys- tifying pseudo-lidar for monocular 3d object detection. arXiv preprint arXiv:2012.05796 2020.
Pei Sun, Henrik Kretzsehmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Com- 2020.
puter Vision and Pattern Recognition, pp.
Yan Wang, Wei Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Wein- berger. Pscudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8445 8453, 2019.
Xinshuo Weng and Kris Kitani. Monocular 3d object detection with pseudo-lidar point cloud. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 0-0, 2019.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018
Zetong Yang. Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1951-1960, 2019.
Sergey Zakharov. Wadim Kehl, Arjun Bhargava, and Adrien Gaidon. Autolabeling 3d objects with differentiable rendering of sdf shape priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12224-12233, 2020.
Wu Zheng, Weiliang Tang, Li Jiang, and Chi-Wing Fu. Se-ssd: Self-ensembling single-stage object detector from point cloud. arXiv preprint arXiv:2104.09804, 2021.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. arXiv: /904:004440000 2019.
Objects as points. arXiv preprint
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4490- 4499, 2018.
12

---

Published as a conference paper at ICLR 2022
Appendix
Due to the space limitation, provide details omitted in the main text in this appendix, which is organized as follows:
Section
Network and the running time.
Section
Detailed ablation studies.
Section
Performance on Waymo.
Section
Generalization ability.
Section
Details of object-LDDAR-ponns
Section
Qualitative results.
Section
Orientation analysis.
Section
Statistics on 3D object labels.
Section
Discussion on viewpoint differences.
Section
Discussion on difficult cases.
Section
Discussion on different baselines.
Section
Results on NuScenes.
A NETWORK AND RUNNING TIME
For the feature encoder, we adopt different backbones includ- ing ResNet18, 8, ResNet34, and ResNet50 He et (2016). We train our network with a batch size of by default. Specif- ically, remove the output layer and the last convolution stage of ResNet, and obtain 7 x Rol Aligned al. (2017) object features from extracted deep features. As shown in Figure 7 we reshape each output tensor to a 1-dim tensor, passing fully connected layers regress corresponding out- puts. We show the quantitative results on different backbones Table 7 Under the APBEy/AP3D (IoU=0.3) metric, dif- ferent backbones all show satisfactory and comparable accu- racy, demonstrating the robustness our method. If using the stricter AP (IoU=0.5) metric, a better backbone is more beneficial for the overall performance as it encodes richer features for object 3D localization.
Object Features
Output
Reshape
Ngx256 KgX0,
TG-102467H7
Ngx50176
Figure 7: Regression network. n4 denotes the number of objects, and no denotes the number of required 3D box parameters.
Table 7 shows the rumning time for the forward pass at different backbones. Thanks to lightweight design, the time-cost is little. The adopted 2D detector takes extra 60ms in inference, meaning that the overall running time of our method is less than 80ms, meaning that method is efficient.
Table 7: The experimental results different backbones on KITTI validation set.
Backbone
Time (ms)
APBEv/APsp (loU=0.3)
Rao
AP APBev/l APpev/APap (loU=0.5) Rao
Easy
Moderate
Hard
Easy
Moderate
Hard
ResNet18
5.90
75.53/70.87
56.45/51.20
46.83/42.33
50.92/44.78
33.17/25.57
26.23/20
ResNet34
8.77
80.92/77.66 59.55/55.10
48. 18.69/44.85
57.15/48.39
36.00/27.14
27. 27.90/21.53
ResNet50
11.97
81.17/78.44 59.87/56.42
48.98/45.81
58.20/50.16
38.02/29.94
30.17/23.11
B DETAILED ABLATION STUDIES
We conduct extensive experiments to validate the effectiveness of each component in WeakM3D. As shown in Table g the baseline (Experiment using only the center loss shows extremely low
13

---

Published as a conference paper at ICLR 2022
Table 8: Detailed ablation study. "Disten." in the table refers the learning disentanglement, and "Geo, align." is the geometric alignment.
APBEv/APaD (loU=0.5)) R4o
Exp.
Disten.
Geo. align.
Ray tracing
Loss balancing
Easy
Moderate
Hard
0.11/0.08
0.16/0.10
0.15/0.04
16115/5
11.98/8.16
9.21/6.04
49.09/37.79
24.86/17.73
18.24/12.60
32.39/24.79
14.56/10.34
10.67/7.38
0
v
0.77/0.26
0.45/0.13
0.28/0.11
V
51.36/45.04
33.08/26.34
26.42/21.19
8
49.92/41.91
32.90/25.24
27.32/20.50
V
17.79/12.75
12.04/7.38
9.03/5.60
V
V
53.02/44.12
34.53/27.07
26.96/21.90
j
V
51.56/44.11
33.77/27.05
26.73/21.10
k
V
V
51.44/43.79
32.07/26.29
25.86/20.69
V
V
55.31/48.35
36.28/28.48
29.03/22.18
m
V
52.50/47.58
33.45/27.93
26.31/21.60
V
56.25/49.78
34.34/27.30
26.82/20.76
56.60/49.03
37.37/29.43
28.68/21.77
V
58.20/50.16
38.02/29.94
30.17/23.11
detection accuracy, indicating the difficulty for the weakly supervised manner. When only employ- ing one of the components, most improvements, as enables the network to learn the most 3D information. When employing two of the components, can know that the combination of geometric alignment and ray tracing (Experiment i) performs the best. Such two strategies mutually reinforce each other, allowing the network effectively learn the object's 3D location. Also, after applying loss balancing (Experi- ment o), unevenly distributed point-wer wait-wise losses can be balanced to facilitate the learning process. Finally, if we use all proposed strategies (Experiment p), we can achieve the best performance. Fur- thermore, instead of freezing the object dimension, we let the network learn the dimension and show the results in Table 9 learning burden, thus making the network perform slightly worse. Our detailed ablation studies comprehensively show the power our method and how such strategies interact.
observe that the geometric alignment (Experiment c) brings the
We can see that adding more entangled learning objectives brings heavier
Table Ablation study for dimension.
AP APrey/AP3D (IoU=0.5) Rio
Dimension
Easy
Moderate
Hard
Learned
56.24/48.37 58.20/50.16
37.04/29.41
29.45/22.74
Frozen
38.02/29.94
30.17/23.11
C PERFORMANCE ON WAYMO
We also evaluate our method on Waymo Sun et al. (200)) dataset, which a considerably large dataset, and provide the results in Ta Waymo dataset provides many annotations, thus the fully supervised manner method M3D-RPN Brazil Liu 2019) is supposed to perform much better than method. However, to M3D-RPN. Especially for medium and far objects, our method outperforms it with a significant margin. It indicates that our weakly supervised manner is more suitable massive data. Also,
observe that our method obtains comparable performance compared
14

---

Published as a conference paper at ICLR 2022
Table 10: Comparisons Waymo. We compare method with the fully supervised method
M3D-RPN. We obtain comparable performance.
APBEv (1OU=0.5)
APaD (IOU=0.5)
Difficulty
Approaches
Overall
0-30m 30-50m
50m-o0
Overall
0-30m 30-50m
50m-ox 00
M3D-RPN
8.01
24.04
3.92
0.40
5.87
19.22
2.40
0.19
LEVEL 1
WcakM3D
8.72
20.39
7.73
1.14
4.81
12.20
3.78
0.46
M3D-RPN
7.51
23.96
3.81
0.35
5.50
19.15
2.33
0.17
LEVEL 2
WeakM3D
8.18
20.31
7.50
0.99
4,50
12.16
3.67
0.40
Rew LIDARt Point Cloud
Clomeer
Cluster
Using 2D Masks
Cluster
or Using 20 Boxes
LijA points
RANGAC
Initaal Rot LIDAR
Clustered pointa
20
LDAR Overground
Figure 8: Details of obtaining Liacct-LiD---ppol Best viewed in color.
prior point-cloud- -ased weakly supervised methods and many fully supervised monocular methods do not evaluate their results Waymo, thus method sets good baseline for future works.
Notably, this comparison, the performance WeakM3D is still limited by the number of training samples. Such samples can be easily collected in real applications, we do not require the extra manual annotating process. In other words, our method is inherently suitable for the autonomous driving scenario, which automatically produces more and more unlabeled data, therefore it has much potential to be explored.
D GENERALIZATION ABILITY
Our method robust and works well for most object categories. We follow a basic assumption adopted by almost all 3D detection works, i.e. all objects in 3d detection are represented cuboids. Please consider the underlying mechanism of human an- notating. People annotate each object by finding reason- able minimum 3d bounding box. Correspondingly, our loss functions exactly aim learn this reasonable mini- mum 3D bounding box for the object. Our method there- fore works well for most object types. To demonstrate this, we conduct experiments on pedestrian and cyclist cat- egories in Table 11 and we obtain promising results. To the best of our knowledge, we are the first weakly super- vised method that works for the two classes.
Table 11: Performance other cate- gories.
APBE (lo==0.25) Rao Moderate
Categories
Easy
Hard
Pedestrain Cyclist
3.79
3.21
3.12
5.16
3.60
3.33
E DETAILS OF OBJECT-LIDAR-POII
As mentioned the main text, we can obtain an initial object point cloud by using the raw LiDAR point cloud and 2D boxes/masks. This initial point cloud contains some noisy or background points, which should removed. As shown in Figure g we cluster the initial object point cloud using an unsupervised clustering algorithm Ester et al. (199)). in which points are divided into different clusters in terms of the density. Therefore, we select the cluster with most points as object-LiDAN points and remove other points. In particular, Line----iiRsss....... points. These inner points from the bird s-eee-view perspective do contribute the object's 3D location. Then, randomly sample 100 object-
further remove the points if their coordinates ysd
are lower than the median object-LiDDe LiDAR
15

---

Published as a conference paper at ICLR 2022
Figure 9: Qualitative results. Best viewed in color.
LiDAR- LiDAR-points. Such points precisely describe a part outline of an object 3D space, and they are ideal weak supervisions for corresponding 3D box prediction.
F QUALITATIVE RESULTS
We provide qualitative results in KITTI validation set in Figure We can that our method works well for most cases, while can fail for the hard cases, egg, truncated, heavily occluded, and far samples. This inspires to further improve the performance in future work.
G ORIENTATION ANALYSIS
Loc.
In monocular 3D detection, the orientation con- sists of two types of orientations, i.e., the global orientation 0y and the local orientation 5y. where the local orientation is also named the observation angle. We show the relation- ship between 0, and 6, in Figure [0] For monocular 3D detection, the global orienta- tion By is hard to reason directly. Even hold- ing the same global orientation 0y, the ap- pearance of object shown on the image can change as the viewpoint varies. We give example in Figure 10 For the location 1, 2, and 3, the mainly captured visible sur- faces by the camera different, consequently
Lac.
Main visible surface
Main visible strface
Main visible surface
Camera
Figure 10: Local orientation 8y and global orien-
tation 8y. The same object with the same global orientation at different viewpoints shows different appearances the image.
16

---

Published as a conference paper at ICLR 2022
Chets
Chick
OteeA
Figure 12:
Distribution of directions of paired points. The red line the box refers to the mainly captured surface, meaning that LiDAR points captured from this surface dominate the object- LLDAR-points A low offset de refers to the orientation close to T/2 while high offset indicates the orientation close to or π. Best viewed in color with zoom in.
showing varied appearances on the image. In light of this, instead of directly regressing the global orientation By. we estimate the local orientation bg and then convert it to dy by tak- ing the camera viewpoint into account. More details can be found in Mousavian et al. (2017).
Also, we provide details for obtaining the global orientation from Object-LiDAR-pont -pintts. Object-LiDAR points denote a part outline of the object, which is highly related to the global orientation Oy- Specifically, we calculate the direction of each pair of object-LiDDAR points and then draw the corresponding histogram. We show the paired points and the histogram Figure [] and Figure 12, respectively. The direction ay that highest in histogram refers By the perpendicular direction of fy. To further determine the orientation, we use the spatial offset d₂ of object-LiDAR-poiie usually faces forward, we first convert ay to the range (I 3x subtracting 5. thus 0y is derived follows:
Pa
Pq
Figure 11: Illustra-
tions of paired points. connect each pair of points and then calculate the direction of the re- sulting line.
along x axis in 3D space. Since car in motion
4 by adding
Og
T
if dg > C and 0g 2 5
By
Ogy
+
if du > C and 0gy < 2
(6)
ay
otherwise.
where C' is the offset threshold. C is set to 3.0 meters by default. From Figure 12 and Equation 6 we can know that a low offset indicates the orientation close to π n/2 while high offset denotes the orientation close to 0 or π..
H STATISTICS ON 3D OBJECT LABELS
We conduct statistics on 3D object labels for the car in KITTI Geiger et al. (2012) and show the results in Figure 13 Please note, we do priors. Specifically, we draw the histogram for the object dimension and 3D location. As mentioned the main text, objects that belong the same class usually have close sizes. We can observe that the height of cars mainly varies from 1.4 to 1.6 meters. Similarly, the width of cars mainly varies from 1.5 1.7 meters, and the width of cars mainly varies from 3.3 4.5 meters. In contrast the low-range varied dimensions, the object's s 3D locations vary lot, especially for (rad, 2sd), e.g., the object depth (23d) varies from to 80 meters. Considering the IoU (Intersection over Union) cri- terion and the ill-posed nature of monocular imagery, improving the localization accuracy is much more important than obtaining more precise object dimensions. The satisfactory performance of our method also indicates that the manner of taking a fixed object dimension is acceptable for weakly pervised monocular 3D detection. Intuitively, in future work we can attempt to obtain more accurate object dimensions to further improve the accuracy.
the statisties 3D box labels the dimension
17

---

Published as a conference paper at ICLR 2022
1620
2903
29221
Numbats
Numbers
502
200
200
322
3.4
1.6
1.8
20
22
24
32
1.5
2.0
25
3.0
33
4.0
50
Hleight (medees))
Width (maters)
Length (melers)
(a) Object dimension distribution.
2982
3220
3000
Numbers
Numbers
600
1500
400
350
206
Location-X (meters)
(metton))
Location (metees)
(b) Object location distribution.
Figure 13: Statistics on 3D object labels on KITTI.
LEAR
Case 1 (cannot exist):
Case
l(camera)
(UDDAR
4(camers)
(LIDAR)
Surface 3(camera)
Surface (UDAR)
Ego-cardi co0-ca/robol
Surface a(camera)
Surface (LIDAR)
Surface 2(camera)
Surface 4 (LIDAR)
Surface 3(camera) Surface 4 (UDAR)
Surfoce
Objjcns
Surtaca
Surfaci
Surfazu
Mismatches (have small impact)
Durfact
Burieee 5
Surlace
Algeing
Theoretical failure cases caused by viewpoint differences between the LiDAR and camera. A well known yet important fact is that the LiDAR and camera are installed in the same car/robot, which indicates that their viewpoint differences are marginal compared the extensive 3D space. We summarize the theoretical failure cases caused by the viewpoint differences. For case 1, it cannot happen since the camera and LiDAR always face towards objects. The occlusion constraints for surface 1 and 3 are the same for the camera and LiDAR. For case 2, the object width W Wobject lower than W Wcom->tide
Figure 14:
(the offset between camera and LiDAR along axis) meanwhile the few. Also, the
object center is very close to the ego-car/robot center along X axis. Such objects adverse impact brought by this type of mismatching can be ignored since Wasject is very small. For case 3, the conditions
similar case 2, but the object can have bigger W Wobject- In this case, the LiDAR mainly captures points from surface 3, such points dominate the loss since points captured from surface
very few. Thus mismatches on surface 2 can also be ignored in the overall loss. Best viewed in color with zoom in.
18

---

Published as a conference paper at ICLR 2022
I DISCUSSION ON VIEWPOINT DIFFERENCES BETWEEN LIDAR AND CAMERA
In the real world, the LiDAR and camera have different viewpoints, which may have some impacts on the proposed losses. Therefore, in this section we discuss the impact brought by viewpoint differences. First, if an object exists in one sensor while invisible in the other sensor, we cannot obtain the associated object-LiDAR- -pontss. In this case, we directly ignore this object. Second, if an object is visible in both camera and LiDAR, the viewpoint differences may bring some impacts in some conditions. For this case, we provide detailed discussion Figure 14 In sum, we can ignore the impact brought by viewpoint differences.
J DISCUSSION ON DIFFICULT CASES
Potential noises and sparsity can exist in clustered points from faraway and small objects. Such faraway and small objects are typical dilemmas for most 3D detection methods due to the inadequate visual clues on the image or few LiDAR points. Unfortunately, our method is general design and does not introduce special components for such objects. We may use depth completion to enhance the raw LiDAR point cloud, to alleviate this issue. This solution can be explored in future works.
Also, when only one object surface (e.g., only the back of a car) is observed, the network cannot learn the complete object dimensions, and the learning of object locations is also affected by object dimensions. In other words, such LiDAR point clouds only provide part of weak supervision, thus the network trained with them performs suboptimally compared to using more complete LiDAR point clouds. Fortunately, not all LiDAR point clouds are incomplete, and other good LiDAR points provide accurate and complete object 3D information, including object 3D locations and dimensions. Therefore, the network still works well after being trained with different types of LiDAR point clouds.
Table 12: Comparisons on different supervised manners on KITTI validation set for car category when using the our baseline network.
AP APRgy/APsp (loU=0.5) R40
Supervision
Easy
Moderate
Hard
Full
53.61/44.78
29.59/23.90 38.02/29.94
34.84/28.66 64
Weak (Ours)
58.20/50.16
30.17/23.11
Table 13: Comparisons different supervised manners KITTI validation for car category when using PatchNet Ma
22200) baseline.
AP APpev/APap (IoU=0.5)) R4o
Supervision
Easy
Moderate
Hard
Weak (Ours)
59.41/54.10 64.31/57.94
37.01/32.47 39.45/35.25
28. 28.53/24.27 34.76/30.07
Full Ma et al. 2020
K DISCUSSION ON DIFFERENT BASELINES
It is interesting to see the performance gap between the fully supervised and our weakly supervised manner on different baselines. For the proposed baseline network in the paper, we report the results in Table 12 Interestingly, on this baseline network, we can see that our method performs better than using the full supervision. This tendency is the same as Table since our method takes more geometry and scene constraints into consideration, benefiting simple baselines. On the other hand, when using more advanced baselines, our method performs worse than using the full supervision (e.g., WeakM3D-Patte Patchhett usefulness of geometry and scene constraints of our method. To comprehensively investigate the performance advanced baselines, report the results PatchNet Ma et al. (2020) under the
PatchNet in Table []. It is because advanced baselines weaken the
19

---

Published as a conference paper at ICLR 2022
Table 14: Results NuScenes validation the category. We provide baseline for future works.
Category
AP
ATE
ASE
AAE
Car
0.214
0.814
0.234
0.682
IoU 0.5 criterion in Table 13 We can observe that the advanced baseline can adequately use the full supervision, thus showing better results than method.
L RESULTS ON NUSCENES
To encourage future works on weakly supervised monocular 3D detection, we further provide results on the NuScenes dataset, which can serve as a baseline for future works. We report the results in Table 14] We use the pre-trained Mask-RCNN, to obtain Rol LiDAR points in training, and to obtain 2D detections inference. Please note that the pre-trained Mask-RCNN cannot output all classes the category. Additionally,
annotated in NuScenes, we only conduct experiments report the AVE (Average Velocity Error) and AOE (Average Orientation Error) because we can not obtain supervision of the velocity and the moving direction the weakly supervised method.
do not
20


Reviews:
[{"review_id": "ddZ6gZUo_o", "reviewer": "Reviewer_D4LN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper is the novel method for the task that has not received much attention but is important. The proposed method is simple and intuitively makes sense for cars in driving scenes that is the most important 3D object category for detection. The paper is well written and the experiments clearly show that each of the proposed components is effective (Table 5).\n\nHowever, I think the novelty of the task might be limited as an important paper is missing from the reference. The paper is \"Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors\" by Zakharov et al. presented at CVPR 2020. This paper addresses the task of learning a 3D object detector from unlabeled images and point clouds.\n\nThe proposed method is different from [Zakharov et al. 2020], but it is unclear which is better. However, it is unclear which is better, because the differences are not discussed and no experimental comparison are provided. The percentage of performance drop (10%~20%) due to the elimination of supervision seems to be almost the same in the two papers, but direct comparison is not possible due to the different baseline methods and evaluation metrics. This submission would be much strengthen if an experiment to compare with [Zakharov et al. 2020] is included. I think discussion on the differences in assumptions and conditions would needs to be included.\n\nI am also concerned that the number of object categories to which the proposed method can be applied may be small. The proposed loss function assumes that point clouds are located near the edge of a bounding box. This assumption holds for a rectangular object such as a car. However, it does not for many other categories such as pedestrians and cyclists, and I suspect that the proposed method would rather worsen detection performance. It is fine that it can only be used for cars, but I think it would be better to clearly state so.\n\nI think the optimal 3D location that minimizes the loss function can be done as a preprocessing step, similar to what is done for rotation angles. I am interested to see how the performance changes with this \"autolabeling\" setting. I guess it will make training stabler. (This is just an interest, not a request for additional experiments.)\n\nOther concerns\n\n- The results in Table 3 seems strange because the accuracy with weaker supervision is consistently higher than that with full supervision. Maybe the first and second rows are swapped?", "summary_of_the_paper": "This paper addresses the task of learning monocular 3d object detection from unlabeled images and LiDAR point clouds with the help of off-the-shelf 2D detection and segmentation models. This task is usually performed by predicting the 3D location, 3D size, and 1D rotation of an object in a 2D bounding box (BB). Since no 3DBB labels are provided, the authors propose to fix the size to a predetermined value and use heuristically estimated rotation angles from the point cloud as supervision. For 3D location, the authors propose a novel loss functions. The assumptions are that (1) the points belonging to an object are located near the surface of a BB and (2) the surface of a BB with points is visible to the camera.\n\nThe experimental results show that the proposed method can eliminate the need for 3DBB labels at the expense of 10~20% lower performance of existing detectors (PatchNet in Table 1 and CenterNet in Table 3).\n\nThe claimed contributions are (1) the novelty of the task, (2) a novel method for it, and (3) its good performance on the KITTI benchmark.", "summary_of_the_review": "This paper proposes an effective method for an important but under-addressed issue. However, the novelty of the task is limited and the comparison with prior methods does not seem sufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SQpU1oBwJGm", "reviewer": "Reviewer_1Zpn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Strength ** \n- Clear motivation and good problem. Addressing the annotation problem for monocular 3D object detection is a task valuable in practice. \n\n- The experiments are extensive and the results are encouraging. While the results are not outperforming all supervised methods, they do outperform previous weakly supervised methods and some previous supervised methods. \n\n** Weakness ** \n- The proposed method does not estimate 3D box parameters fully. It relies on priors to estimate object sizes. As shown in the appendix, the object size is estimated by computing statistics from the annotated dataset.\nIn fact, in my opinion, to be fair, the results of learned dimensions should be shown in the paper, and the results of the frozen dimensions could be shown as a supplementary result. \n\n- Section 2.1 mentions a few methods that are 'widely employed in autonomous driving'. Can this claim be supported by any evidence or citation? While LiDAR-based methods have higher accuracies, their adoption in the autonomous driving industry was not well documented. \n\n- To avoid confusion, please indicate at the table caption that the experiments are for Car class. E.g., in Table 2, 3 this is not mentioned.\n\n** Questions ** \n\n1) Both geometric alignment loss and ray tracing loss assume that if we take the bounding box of the observed LiDAR points, it should somewhat have similar sizes to the ground truth box if the annotation is done. \nHow do these two losses perform in the case of amodal detection where let's say, only the back of a car is observed? How well is the alignment work? \n\n2) LiDAR point clouds can be noisy, i.e., the points observed across frames are inconsistent. How does this affect the weak supervision? \n\n3) The weak supervision relies on preprocessing such as ground plane estimation and clustering, which imposes some limitations.  \nIn my understanding, if clustering works well in the beginning, we might not need detection anymore. If clustering does not work, the weak supervision could be meaningless. Could you elaborate on which cases such preprocessing fails? \nDoes it make sense to regress boxes from the point clustering and compare to the ground truth boxes to see how well the clustering is? \n\n4) The proposed method outperforms some previous methods in KITTI validation set, but does not outperform on KITTI test set, e.g., see RTM3D method on the KITTI leaderboard. And this is not shown in Table 2. Why is there such a performance gap? \n\n5) The metrics used for experiments are rather inconsistent. I suggest reporting both AP11 and AP40 when suitable, and keeping all IoU to 0.7. I am confused about the switch from AP11 IoU 0.7 (Table 1) to AP40 IoU 0.7 (Table 2), and then to AP40 IoU 0.5 in Table 4 and the ablation study. \n\n6) An inherent limitation of the method is the handling of occlusion. Due to viewpoint differences, some objects might appear in the LiDAR view but not in the image view and vice versa. How did the authors handle this problem? \n\n7) Did the authors evaluate with nuScenes dataset? As far as I know, more papers report their performance on this dataset than Waymo. Also, this dataset also comes with evaluation metrics for translation, orientation, separately, which could provide more insights. ", "summary_of_the_paper": "The paper presents a new method for weakly supervised monocular 3D object detection. The authors are motivated by the fact that 3D bounding boxes for monocular video are expensive to annotate, and so propose to use 2D boxes and RoI on LiDAR point clouds to supervise the detector by minimizing the alignment between the RoI on LiDAR and the 3D box prediction. The results show that the proposed method achieves good performance on the KITTI and Waymo dataset. ", "summary_of_the_review": "The paper presents interesting ideas for training monocular 3D object detectors with just LiDAR point clouds supervision, and the proposed solution is reasonable and achieves decent performance. Therefore I am leaning a little positive despite there are several concerns that I invite the authors to address in the rebuttal. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ndb_cRgQfUw", "reviewer": "Reviewer_Pqqk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This review points out the strengths and weaknesses of the paper in descending order. I have gone through the methods, technical implementation, and experiments in detail. In my opinion, the proposed method is sound under the specific setting of this paper.\n\n**Strengths:**\n\nI like the idea of this work mainly from two aspects:\n1) This work proposes an idea that is straightforward but (seems) has not been explored in the community: Lidar points can provide weak supervision to estimate 3D object bounding boxes. The nature of Lidar makes the captured points close to object edges, which could briefly describe the shape. Meanwhile, the sparsity of point clouds largely reduces the cost to recognize background points. These two properties make this work possible.\n\n2) This work notices the problem that Lidar points usually only exist in one edge since it captures scenes in one observation. It proposes a ray tracing loss to mimic the occlusion problem and encourage the network to predict 3D boxes that put ‘object-LiDAR-points’ to the edge toward the observer. This regularization term does make sense and effectively boost the detection performance. But by the way, ‘semantic ambiguity’ seems not to be an appropriate term to describe this problem, and is actually a bit distracting. It may lead people to think about object categories or something close.\n\nThe design of other components, such as geometry loss, also provides a complete training framework.\n\n**Weaknesses:**\n\nMajor:\n1) The derivation of geometry loss (Eq 1) and ray-tracing loss (Eq 2) relies on building virtual rays from a camera point. It assumes that the distance between the camera and Lidar and their observation angle difference are small enough to ignore, which is not always true in the real world. Rectification will not solve this problem because the observed planes may be different. I was hoping the authors could provide further discussion on this point, e.g., how could their method work in this setting or why the potential drawback is negligible.\n\n2) It is good that the authors have noticed that Lidar points often gather along edges, and proposed a ray-tracing loss. However, this work did not explain or try to analyze how the network could predict a complete 3D bounding box, when only one object edge is constrained. For example, how would the network predict the length of a bounding box if all the Lidar points stay at a corner? In my humble opinion, the network may highly rely on the templates (or say priors) of height, width, and length (Sec 4.1). This might be supported by the observation that the proposed method performs well on easy metrics while has a clear gap against other methods on stricter metrics like AP moderate or hard, as shown in Table 1.\n\nMinor：\n\n3) The table captions are confusing, especially those on Page 8 and 9. For example, in Table 6, the caption is just ‘Comparisons on different requirements’. Table 3, ‘Comparisons on CenterNet’. Nothing meaningful is provided in such a short sentence, and the authors did not provide further explanations. Moreover, in Table 6, the Requirements are ‘2D box + LiDAR’ or ‘2D mask + LiDAR’. But in my understanding and the content of other sections, the proposed method does not need Lidar during inference. Although I could understand the meaning, the authors should provide comprehensive descriptions in the Section Experiment.\n", "summary_of_the_paper": "This work, named WeakM3D, notices that Lidar points can provide weak supervision for monocular 3D Object Detection, dispensing with expensive 3D box annotations. It further identifies some challenges when trying to use Lidar points as supervision, and proposes several losses to mitigate the problems. Quantitative experiments on various datasets show that the method could match some fully supervised 3D object detection methods. I believe this paper is novel to some extent and has merits, though some of its claims need further support.", "summary_of_the_review": "Overall I like some ideas of this work but it seems the authors should provide more analysis to support their claim. More discussion on how the proposed supervision works is also beneficial.\n\nOverall I think this paper is a good starting point. I may raise the score if the authors could solve the concerns on loss derivation (weaknesses 1) and provide analysis on complete bounding box prediction (weaknesses 2). I guess weaknesses 2 cannot be fully avoided, but a detailed analysis can back up the authors’ claims much better.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6abrhk_0CS7", "reviewer": "Reviewer_Tcr1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The motivation is good. This method uses object-LiDAR points to supervise the 3D prediction of monocular images. Such practice can solve the problem of high-cost 3D annotation.\n+ The writing and organization of this paper are good. The challenges described in the introduction are well-addressed by proposed solutions. \n+ The idea is easy to follow. Both the main components and details are clearly explained. \n+ Experimental results demonstrate the effectiveness of the proposed method. \n\nWeakness & Questions\n- The proposed method has strict requirements on the sensor configuration. The LiDAR and camera need to be well-calibrated and synchronized. Such requirements limit the generality of the method. \n- The key component of the weakly supervised training is the selection and filtration of object-LiDAR points. Although the authors use ground-plane subtraction, 2D-box selection, and unsupervised clustering. The quality of object points is still questionable, especially the long-range and small objects. How to relieve the problems of potential noises and sparsity in such clustered points?\n- The formulation of Equation 3 is not very accurate. The $\\varepsilon$ is the number of LiDAR points in the neighborhood but not the density. The lower part of the summary operator is suggested to add $j!=i$. \n- In the learning disentanglement, the dimensions of objects are directly indicated by prior knowledge. However, the sizes of objects in the same category are very different. For example, the length of the vehicle varies from 3m to 5m. Is that reasonable to use a single value to represent the whole category? How about the average scale error (ASE)[1] of final detection?\n- The authors report the running time for the forward pass in the Appendix. What is the time cost in the training period since it is involved with the large computation of ray-tracing. (This point is just a question but not weakness.)\n\n[1] Caesar, Holger, et al. \"nuscenes: A multimodal dataset for autonomous driving.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n", "summary_of_the_paper": "This paper introduces object-LiDAR points to support weak supervised monocular 3D detection and obtain good results.", "summary_of_the_review": "This paper proposes a method named WeakM3D to use object-LiDAR points to supervise the 3D learning from monocular images. In order to align the object-LiDAR points with monocular 3D predictions, the paper analyses the challenges and presents corresponding techniques or loss constraints. The method can train the models without manual labels, which is practical for monocular detection tasks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5p5iHKkUHM", "reviewer": "Reviewer_fzTD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Novelty of addressing the weakly supervision: two introduced losses are reasonable for weak supervision from raw point clouds.\n+ Organization of the paper is clear and easy to follow.\n\nWeaknesses:\n- The biggest concern is the motivation of the problem statement. When the LiDAR point clouds are available, why not use 3D point cloud based object detection as the supervision for the monocular object detection task? I suggest the authors show using the proposed method can achieve better performance than using the SOTA 3D object detection results as supervision. \n- Reference Meng et al. (2020) is an important related work, which is also mentioned in the manuscript. But seems the authors are not doing any comparisons between their work and this reference. Any reasons?\n- The paper names the backbone network as a 3D feature encoder, but seems the backbone is just a classical 2D image backbone without any 3D mechanisms (e.g., depth-aware convolution, etc.). I would suggest the authors try some more advanced backbones designed for 3D object detection, which may further improve the performance. \n", "summary_of_the_paper": "This paper proposes a weakly supervised monocular 3D object detection framework using LiDAR point cloud as the supervision instead of 3D bounding boxes. In order to achieve this, the authors introduce geometric alignment loss and ray tracing loss to address point cloud alignment, loss balancing to address point cloud density issues, and a training strategy for learning disentanglement. The paper is overall well-written and clearly stated. However, the motivation of the problem is questionable and is not clear to me. Please see the detailed comments below.", "summary_of_the_review": "Overall, this paper is in good shape. The most critical concern is motivation. If the authors can provide a reasonable explanation on the motivation as well as other concerns mentioned above, I would like to consider recommending acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 5 quality reviews. The rebuttal and discussion were effective and addressed many concerns from the reviewers, after which most reviewers increase their ratings of this paper. The final rating is 6 from 4 reviewers, and 8 from 1 reviewer. The AC concurs with the positive recommendation from the reviewers and recommends acceptance.
Decision: Accept (Poster)