Depth-discriminative Metric Learning for Monocular 3D Object Detection
Authors: Wonhyeok Choi
Mingyu Shin
Sunghoon Im
Venue: NeurIPS 2023 poster (2023)
Keywords: Monocular 3D object detection
Autonomous driving
Recognition
Regression
Metric learning
TL;DR: A novel metric learning approach to improve monocular 3D object detection without increasing model complexity.

Abstract: Monocular 3D object detection poses a significant challenge due to the lack of depth information in RGB images. Many existing methods strive to enhance the object depth estimation performance by allocating additional parameters for object depth estimation, utilizing extra modules or data. In contrast, we introduce a novel metric learning scheme that encourages the model to extract depth-discriminative features regardless of the visual attributes without increasing inference time and model size. Our method employs the distance-preserving function to organize the feature space manifold in relation to ground-truth object depth. The proposed $(K,B,\epsilon)$-quasi-isometric loss leverages predetermined pairwise distance restriction as guidance for adjusting the distance among object descriptors without disrupting the non-linearity of the natural feature manifold. Moreover, we introduce an auxiliary head for object-wise depth estimation, which enhances depth quality while maintaining the inference time. The broad applicability of our method is demonstrated through experiments that show improvements in overall performance when integrated into various baselines. The results show that our method consistently improves the performance of various baselines by 23.51\% and 5.78\% on average across KITTI and Waymo, respectively.

Depth-discriminative Metric Learning for Monocular 3D Object Detection
Wonhyeok Choi"
Mingyu Shin"
Sunghoon Im
DGIST, Daegu, Korea
(smu06117, alsrb4446, bunghoooimmOdgiist.. ac.
Abstract
Monocular 3D object detection poses significant challenge due to the lack of depth information RGB images. Many existing methods strive to enhance the object depth estimation performance by allocating additional parameters for object depth estimation, utilizing extra modules or data. In contrast, we introduce a novel metric learning scheme that encourages the model to extract disrimniirri features regardless of the visual attributes without increasing inference time and model size. Our method employs the distance-preserving function to organize the feature space manifold in relation to ground-truth object depth. The proposed ( (K. B, -)--uasi- isometric loss leverages predetermined pairwise distance restriction as guidance for adjusting the distance among object descriptors without disrupting the non- non-linearity of the natural feature manifold. Moreover, we introduce an auxiliary head for object-wise depth estimation, which enhances depth quality while maintaining the inference time. The broad applicability of our method is demonstrated through experiments that show improvements in overall performance when integrated into various baselines. The results show that our method consistently improves the performance of various baselines by 25.27 35.2%% % and 4.54% average across KITTI and Waymo, respectively.
Introduction
Monocular 3D object detections 20 22 23 27 has gained prominence cost-effective and easily deployable solution, playing critical role in autonomous driving and robotic navigation systems. Typically, the frameworks consist of a feature extractor and lightweight multi-head architectures that predict the projected centers, depths, bounding box sizes, and heading directions of multiple objects 27 41). Among these tasks, object depth estimation, inferring the distance from monocular camera to the center of an object, is the most challenging sub- task due depth ambiguity. Previous work (23 has highlighted this issue in ablation studies, revealing that substituting object depth predictions with ground-truth (GT) values significantly improves overall performance, while replacing other sub-tasks, such as heading direction and 3D size, does not notably enhance performance. Furthermore, several prior works 3 [8] 22 27 28 37 41 have attempted to improve object depth estimation quality by adding additional modules or introducing new formulations. However, despite the subsequent increase in inference time and model size, the improvement performance remains limited.
from single RGB image [18 20 22
23
26
These observations suggest that monocular 3D object detection heavily relies on object depth qual- ity: nevertheless. conventional methods yield unsatisfactory results due the extraction less- discriminative features for object depth inference. One reason is that the extracted feature involves visual attributes, such as the color, size, and heading direction of objects, resulting in the object
Equal Contribution Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

depth head receiving feature with limited depth discernment. To improve object depth performance, the network should be capable of extracting purely depth-discriminan features that involve es- sential geometric information, irrespective of the visual attribute. A feasible method for extracting the depth-discriminatti features involves using deep metric learning schemes such as contrastive learning [6] 7 [9] or representation learning [5] 34
However, most existing deep metric learning schemes 66 16 39] rely on aggressive two-view augmentation (i.e. Affine transform) to train the distance or similarity metric. This aggressive data augmentation method is hardly leveraged in current monocular 3D object detection frameworks due to the violation of geometric constraints, where horizontal flip and color distortion are the only two methods used in this field for long time [T9]. One alternative way is to learn regression-awe -wweee aware representation by contrasting samples against each other based on their target distance using GT labels [39. However, forcibly arranging the manifold of the feature space using this depth distance metric may negatively impact the performance of the other tasks, because the feature extractor for monocular 3D object detection inevitably produces complex shared representations across multiple sub-tasks. Note that the experiment of negative impact is conducted in Sec. 5.2
To address these issues, we propose a (K, B, ().a-))ii--sommetti loss, new metric learning approach that encourages the network to extract depth ceptt-diisrrmminttie features using object depth labels. Inspired by several manifold learning schemes [29] B3], we locally preserve the neighborhood distances to maintain the natural non-linear manifold of feature space in order to mitigate the negative transfer effect on other sub-tasks. Our approach utilizes the quasi-isometric properties providing a relaxed condition for the distance metric between the depth and feature metric spaces. This enables the model to arrange the feature space with respect to the object depth labels while maintaining the performance of other sub-tasks. Moreover, we introduce an auxiliary head for object- wise depth estimation to further improve object depth estimation. The head component is removed after the training process, ensuring that the inference time remains unaffected and does not experience increase. Experimental results indicate that the proposed method consistently outperforms state-of-the-art baselines across a variety of 3D object detection datasets. Our method has been shown to be compatible with various monocular 3D object detection frameworks [15 20 22 23 27), demonstrating its broad applicability without compromising inference time or increasing model size. The effectiveness of each proposed module is further underscored through comprehensive ablation studies.
Our contributions can be summarized as follows:
We propose a simple yet effective metric learning scheme that preserves the geodesic distance of depth information to feature space.
We present an auxiliary head for object-wise depth estimation, which enhances the depth quality without impacting the inference time, maintaining efficient performance.
Our method significantly enhances the performance of various monocular 3D object detec- tion methods without increasing inference time and model size.
2
Related work
Monocular 3D object detection.
Monocular 3D object detection can be broadly categorized into 22
two types. The first type 18 20
37 41 predicts the localization of objects of interest around an ego vehicle using only RGB images, annotations, and camera calibrations. Most of them are divided into several sub-tasks, with a primary focus on estimating
23
25
based on CenterNet 43. They
accurate object depth. The works 22 and 41 propose the formulations of object depth estimation.
MonoCon (20) leverages the abundant contexts in conventional 3D bounding box annotations to integrate various auxiliary tasks. The second category leverages additional data, such as pre-trained
28 35) or CAD models 24 38), to compensate for the lack of 3D information in monocular images. Some prior works utilize pre-trained depth estimators to address the lack of depth information, either by converting the monocular setups LiDAR/Stereo environments 35) or by allocating more parameters to the depth estimation task [10
models 88 TO
[
27
15 40 Alternatively, certain methods employ completed depth maps from pre-trained depth completion models (27 or pre-trained
LiDAR-based detectors R [4] to supervise their models. Notably, DID-M3D 27 divides object
depth into visual and attribute depth using completed depth maps to address the ambiguity of the object depth.

---

Manifold geometry preservation. Deep learning networks, consisting of continuous and differ- entiable (precisely, almost everywhere differentiable) layers, ensure smooth mapping of networks f I + RR" This guarantees that neighborhoods in IR will be mapped into neighborhoods in the embedding space IR" with some amount of "stretching" and vice versa. Manifold learning has been studied to preserve most or all of the essential information by minimizing these stretches. Many works [1 2 17
29 42] aim to create meaningful representations when embedding high- dimensional data into lower dimensions. The traditional linear embedding algorithms commonly use the matrix decomposition to preserve the original variance [I or pairwise distance [17]. More recent algorithms 2 29
construct the neighborhood graphs by preprocessing the dataset using e-ball or k-nearest neighbors for each point, embedding the data while preserving the neighbor samples' distance. In particular, Isomap B3 estimates the shortest path in the neighborhood graph between every pair of data points and then employs the Fuclidean Multidimensional Scaling (MDS) algorithm [17] to embed the points in dimensions with minimum distance distortion all at once. These non-linear embedding algorithms can account for the non-linear nature of the manifold by preserving the distance between neighborhood samples.
33 42
Metric learning. Metric learning is one of the machine learning techniques that aims to learn an effective distance metric between data points by using training data. To achieve this, methods such as [6 attempt to minimize the distance between samples from the same class while maximizing the distance between samples from different classes. These methods demonstrate improved performance for self-supervised learning by learning in an end-to-end fashion, including two-view augmentation or depending on class labels. These deep metric learning methods typically target classification tasks because the positive/negative pairs defined as belonging the same different classes. More recent works [31 36 39] extensively apply metric learning in the context of regression. These works
either directly use the target distance to contrast samples against each other 33 39
or leverage
representation learning in a semi-supervised learning scenario [3
3 Method
Our final goal is to boost the object depth performance by extracting the cepth-discriminative features without increasing inference time. Similar common deep metric learning schemes 6 7 (3) we propose a loss term that preserves meaningful low-dimensional data structure in the feature space.
3.1 Preliminary
Metric space. A metric space is a mathematical concept that characterizes a set of points and a function that measures the distance between any two points in the set. Formally, metric space be defined as a pair (M, d), where M represents a set and d a distance function on M. The distance function d must satisfies the following axioms [3] for any three points x, y, z M:
1. Non-negativity: d(x, y) 2 0 and d(x, v) = 0 if and only if I = y-
2. Symmetry: d(z, v) d(y, 2) for all x, E M.
3. Triangle inequality: d(x, v) d(z, d(2, v) for all v, 2 E M.
Quasi-isometry. A quasi-isometry is a function between two metric spaces that preserves distances up to constant factor, even though it may locally distort angles and distances. Let be a function from one metric space (M1, di) to another metric space (M2, d2). Q is considered a quasi-isometry from (M1,d) d2) (M2, de) if there exist constants K > 1. B 2 and e 2 such that both of the following properties hold:
1. Væ1, E M1
1 d11, B s (2(1)) ((x2)) & K (21, + B.
K
2. V2 E M2 : En E Mi s.l. dz(z, 2(x)) e.
Quasi-isometry does not necessarily require continuity 4. This property advantageous because other
transformations may not possess it, and most datasets are finite. Therefore, we use these conditions as constraints to ensure that the feature space retains the geometrical information of the depth metric space.
3

---

3.2 Problem Definition
The task of monocular 3D object detection aims to predict both the object class e and the 3D bounding box b for multiple objects within an image I. Recently, CenterNet [43] has become the best practice
for monocular 3D object detection, with subsequent works [8] 20 22 23 27 4T adopting its pipeline. These methods decompose the bounding box b = [X, Y, Z, h, wt, 7 estimation problem
into separate estimations of the coarse projected 3D center vu) the depth of object center 2, the center offset (ōu, ōv), the 3D size dimensions (h, w,,),, and heading direction (i.e., the yaw angle). The 3D object center [X, Y, Z] computed by back-projecting the projected center with the corresponding depth, given the intrinsic matrix of the camera K, as follows:
Ue 4 + ou, Ve = + 8v,
2, Ur. 2, 2,= X.Y. = - [Uee
The networks consist of feature extractor Fo and task-specific heads Gør that produce the feature maps h and the per-pixel output maps o,, where i T {les bus bus tour Lous bus the tws 4,, by respectively, as follows:
h Fo),,, Fo R3 + Cxx''WW
(2)
o, = Gar (h). Gor RCCH' ''xW
where H, W represent the spatial resolution of the image, and C, H', W' denote the channel and spatial resolution of the feature maps, which is downsampled from the image resolution. The final object-wise results or are extracted from the non-learnable function H. given the intermediate per-pixel outputs ör (ferfu fu:fdustuu of Atstt:
) as follows:
OT H(ōT), H: R / xW'
T N
(3)
where the number of detected objects N is not fixed and can be zero if no object is detected.
3.3 Methodology
In this paper, our primary focus is training the feature extractor Fe with the objective of extracting features that enhance the discriminability of object depth. At the same time, we aim to preserve the discriminability of other sub-tasks, such as the projected 3D center, bounding box size, and heading directions. To achieve this, we propose a metric learning method that encourages the network to extract sippt-ddicrrmmiativee features by leveraging object depth labels.
Given MM training images I' where i E - (1,..., M. and N GT objects in the image I' we extract object descriptors p(.-)), where j E J = (1, N'), from feature maps h' using GT coarse projected 3D center (é.s), y(اا) and obtain the corresponding GT object depths 2(1J)) We build a set object descriptors P U(eer, jEJ) ((h.)) and a set of corresponding object depth Z = Uperr, jeJ) (1,)) We then define two metric spaces (Z,di) and (P.d d2d)),, where dj and d2 represent the Minkowski distance in Euclidean space (i.e., LI distance). The finite sets Z and P consist of L elements, corresponding to objects in the dataset (ZZ) = |P| = L = Eigr N Consequently, we can establish a one-to-one function Q between these two metric spaces follow:
2(=)) - Q Z P. 1 E (1,2,..... L.).
Our goal is for the function Q enforce a quasi-isometric between Z-space and P-space by using the properties of quasi-isometry in Sec. 3.1 This can encourage the network to extract depth- discriminative features in P-space by utilizing object depth labels in Z-space. However, enforcing quasi-isometric between the low- low-dimensional Z-space and the high-dimensions the non- linearity of the natural manifold. potentially causing the negative transfer to other sub-tasks. Therefore, we adopt the local distance- preeeervineeegggggg condition of the non oon-ein embedding methods such as Isomap [33 and LLE 29 The revised version of the quasi-isometry condition is as follows:
P-space can damage
(i) U (= E Z|= E Br.c.
(ii) V21 E Z, s.t. V22 U. : di (21, 22) - B < d2((s=)) 2(z²)) & K. di 22) B.
(ii) Vp E P 3z Z. s.t. de 2(z)) Se.

---

P-space
P-space (lxal)
te-ball
8 3m
8.3m
Feature munifold
7.5m
10.m
8m
cobject point Of property-violated object pair (noed be close by)
13.1m
gs
13.1m
O: proeery-vvolated object pair (need he fur away)
R
Z space (immutable space)
propert--stiisiic non-neighhhor object puir
R
Figure I: Illustration of our quasi-isometric loss Egi.
where B5,c is ball with radius e centered around By applying the quasi-isometric properties solely to neighboring samples within the e-ball, the non-linearity of the feature manifold is preserved, and the shortest curve distance between any two arbitrary samples on the manifold (a.ka. geodesic) in P-space is maintained instead of the Minkowski distance. By ensuring these three conditions (i), (ii), (iii) are met for sufficiently small K 2 and B 2 O, we can encourage object features of to be depth-discriminative using their corresponding object depth z'. To achieve this, function Q must satisfy these conditions, which involves training the parameter of the feature extractor F.
(K, B, e)- J-Quasi-isomettie loss. We propose quasi-isometric loss term that enforces the quasi- isometric between the depth and feature metric spaces while preserving the distance among the neighbor data points illustrated in Fig. In particular, the proposed loss term is designed to arrange P-space samples that do not satisfy these conditions. To ensure efficient training within the constraints of limited GPU memory, we use the samples within mini-batch instead of using the entire input dataset as done in previous works 2 29 33 We selectively choose the neighbor objects in mini-batch with respect to their corresponding depth labels, where neighboring object depths within U..
Suppose that objects in mini-batch images I, correspond to the object depths Z₁ and object features Pa on Z-space and P-space, respectively. We can then identify the object feature pair sets P P, that violate the revised property (ii) by transposing it as follow:
121, 22l S e: de P2) $ K d1 22) B, V(pI, P2) E P C Pb Pb. s.t.
(5a)
((L1,22) E P, C P Pb, s.t. |2) 22 S
K di(21, B $ d2(p1, (2),
(5b)
where (21,22) is the corresponding depth pair of the object features (p1, p2). For an object feature pair with distance d2(p1, P2) in P-space, the distance is larger for P. in Eq. 5a smaller for P, in Eq. than the corresponding depth pair distance di (21, 22) with factor of K and the additive constant B. To ensure these property-vin mroerty--vollttng object feature pairs P PT satisfy the property (ii), we propose the (K, B, e) -uuaiis--ssyy loss, which modifies the Normalized Temperature-Scann Cross- Cooss-y loss Xent loss) from 6 as follows:
S+ (pisPs)
Lui =
log St + E E(ppp)EP S- (Pkp,
P
(PHP)))P!
(6)
St(p
Pillr K|zi, 2y1 B)/T),
S Sp P2) = exp( -ap(-(1 12,, 221 B --lp,, PPP...Ppllp)/7
K
5

---

Algorithm 1 (K, Quuasissstssssstttttiiiii loss
Input: h. (1,,),, 24, where É {1,2. n), is the number of objects in a batch.
1: hang <- avg avg__ool_5x550 (h)
# feature map avg. pool
2: initialize P=
3: for do
4: P[i]
havgl's Uz, Us]
5: end for
6: Z
{21,
22.... in that correspond to P.
<pipi>> <pi,2>> <pi,Pa>
7: Mp -
<PriPi> <PniP2>
<pninn> where < a > is the Minkowski metrie ||, bllp-
101,21| |21,02| ... |21,21l
8: Mz +
|EniRel |anital ... |m
9: M + MM KMz B # To find property-violated object features. (Eq. 5a
10: M - R- Mp- B To find property-violate object features. (Eq. 5b
11: (M+); t where 12,, 2y >
i2 (M+ Dif O. 12: (M )is + 0, where 22, 23 > e or i i2 j or (M )i 0.
13: anes" +- exp(-MM /T)
# is temperature term.
14: ancs + (exp(( -MM /T))
15: Lgi = mean( log(anes) canne*/(annss" (anest + anes o))) #
12: div. assert
where is the temperature term, and ||, -lp is p-norm, and K B 2 0 denote the pre-defined hyperparameters that determine the hardness of quasi-isometric property. This loss aims to make the property-violated object features comply with the quasi-isometric property by assuming P as positive/negative anchors, respectively. The similarity metrics S+/- are always positive values in (0, 1] that imply a negative distance gap between the distance metric on Z-space and P-space. These adjustments are necessary for proper alignment between the two spaces, noting that P, is derived from Fo
FI,).
According Eq. [ the estimated object depth é and coarse projected 3D center (û, 6) jointly determine the location of the object 3D center (X, Y 2). However, many existing methods 200 21 23 27 train the depth loss on the local region exactly on the GT center (u, v), which can lead to significant errors in object depth estimation
Object-wise depth map loss.
with slightly inaccurate object center (ū, 0) during inference (i.e., center shifting in the image plane by a few pixels) (23).
To mitigate this issue, train our network with additional auxiliary head for object-wise depth estimation using the structure of [20). Rather than solely providing depth supervision to an object center (u, v). the auxiliary head is trained with depth supervision over the entire bounding box of the
object. We create the foreground object-wise depth map D R 'W' following the policy of [40]. Then, we define the object-wise depth loss by adopting the Laplacian aleatoric uncertainty loss (9] the same as the object depth estimation task, only for the foreground regions:
1
Cobs
1 - 2 log (o*),
D
GF
PED
where 2P, SP and GP are the corresponding GT object depth, prediction, and uncertainty of each pixel, respectively. Note that the additional task head is removed after training, the proposed method does not increase the inference time.
Total loss. The total loss Liotal combines the loss used in 3D object detection baselines Ltaselines the quasi-isometric loss Lgis and the object-wise depth map loss Coty as follows:
Listal Loaseline Api Lyj Aobi Lobj.
where Agi and Aati represent the balancing weights for the quasi-isometric loss and the object-wise depth map loss, respectively. These values are set to 0.5 and 1. This total loss is applied to each baseline in our experiments, which we denote "[Baseline] + Ours". as discussed further in Sec.

---

Table 1: Evaluation results on KITTI validation set. (Blue/Red (posiriveennegaiivee relative performance from baseline.)
Car, Afanpoo I Moderate
Car, Afpsvino Moderale
Extra data
Method
Easy
Hard
Hard
LiDAR
DID-MD 22)
23.93
16.22
13.98
32.65
23.15
19.48
DID-M3D Ours
24.77 (13.%)
17.12 (+5.55)
(42.88)
34.32 (.1..5)
(1
20.64 (1-09))
MonsDLE 23
17.32
14.35
12,22
24.62
20.25
17.75
MoneDLE + Ours
21.31 (+23300%)
16.53 (-15.2%)
13,93 (+140%%)
29.34 (+119.2%)
22.27 (+100.0%
19.20 (+8.2%)
Nome
GUP-Net [2]
22.10
16.17
14.18
30.67
22.79
19.64
GUP + Oux
24.21 (1.59)))
17.82 10.2%)
15.01 (4-5.994)
32.38 (15.0%
23.61 (1 (.6%))
(1-7999)77
MoneCon
23:03
17.84
15.37
32.97
24.13
20.90
MoneCon- + Ours
27.90 (+21.19)
19.43 (-8.9%)
16.92 ++ 10.3%
36.15 (+9.8%)
26.15 (+8.01%)
22.56 (+7.9%%
Additionally, to simplify the implementation of our proposed (K, B, quasi-issmmttii loss, we detail the loss function in A
4 Experiments
The KITTI dataset [12] consists of 7,481 training images and 7,518 test images for official KITTI 3D object detection evaluation and contains three categories: Car, Pedestrian, and Cyclist. For additional experiments, we follow 233 which splits the training images into 3,712 and 3,769 images for training and validation sets, respectively. The Waymo dataset [32 a recently released dataset comprising 798 training sequences and 202 validation sequences, with four categories: Vehicles, Pedestrians, Cyclists, and Signs. We use the split reported in [28]. including 52,386 training and 39,848 validation images, to evaluate performance on the Waymo dataset.
Datasets.
Evaluation Metrics. We adhere to the protocol reported in 12] and (8 for KITTI and Waymo datasets, respectively. The KITTI 3D object detection performance is evaluated by the average precision of 3D bounding boxes (AP3D) R40) with IoU thresholds of 0.7 for Car and 0.5 for Pedestrian and Cyclist. The evaluation is split into three levels of difficulty: Easy, Moderate, and Hard, based on the 2D bounding box height, occlusion level, and truncation. In contrast, the evaluation metrics of the Waymo dataset are based on 3D IoU with mean average precision (3D mAP) and mean average precision weighted by heading (3D mAPH). Each object is divided into one or levels and evaluated at three distances: [0, 30), [30, 50), and [50, oo) meters.
Implementation details. First, we incorporate our method into four different baselines, including CenterNet-based frameworks [20] 22 23 [7 for the KITTI dataset. For the Waymo dataset, we choose (20), as it performs best on the KITTI dataset among our baselines. We primarily follow the experimental setup details (i.e., epochs, optimizer) of each paper 20 22 23 for a fair comparison. We denote the plugged versions of baselines as "[Baseline] Ours". As an exception, we set all batch sizes to 16 for the KITTI and Waymo benchmarks. We retrain all baselines on the KITTI/Waymo validation set with "Car" and "Vehicle" classes, respectively. For the (K, B, -)-quusi-isometri loss, we use K = 1.5, B = 0.5, = 10.0, di (---) = "1 dz(;,)) :) = 1|:2 for all experiments. We provide more detailed experimental setups for each baseline in the supplementary materials.
Experimental Results
5.1 Evaluation Results on KITTI and Waymo Datasets
KITTI dataset. We demonstrate the effectiveness of our method on both the KITTI validation and test datasets. As shown in Tab. [ our method can be widely applied to various baselines and significantly enhances performance by a considerable margin. Particularly, the improvement is more pronounced in models that do not utilize extra data, surpassing models that do (13.1% vs. 3.8%). We surmise that the model trained with additional LiDAR data learns more discriminative features regarding depths than the method without extra data. Notably, the performance of "MonoCon + Ours" exceeds that of models trained with additional LiDAR data. This result illustrates that our proposed metric learning method empowers MonoCon to extract depth-discrininative features without the need for extra data. Similar trends are observable the evaluation with the KITTI test datasets, as shown in Tab. 2 The experiments demonstrate that our proposed method enhances depth discrimination not
7

---

Table 2: Evaluation results on KITTI test set.
Car, A T
Ped. Ro
Cyc. APadiro
Extra data
Method
Easy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
DID-MED 2+
24.40
16.29
13.75
LIDAR
DID-MEID Ours
27.04
16.42
13:37
14.41
9.05
8.0
4.86
3.11
2,97
Improvemmnt
(+108%)
(+0.8%)
(-2.8%)
MoneDLE 23
17.23
12.26
10.29
9.64
6,55
5.44
4.59
2,66
2,45
MonoDLE
12.72
11.75
7.80
6.29
6.02
4.12
3/4
Improvemmnt
(+28.3%)
(+21.8%)
(+2366)
(+21.9%
(+19.1%)
(+15.6%)
(+31.2%)
(+31.9%)
(+39,6%)
GUPNet 22 GUPNec
20.11
14.20
11:77
14.72
9.53
7.87
4.18
2,56
2,09
None
13:02
14.23
9.03
8.06
3.13
Inpinwvoenn
(+15.389
(+11 199
(+10669)
(-3.5%)
(-5286
(42.44%
(+75.00)
(4-1.08)
(+-49.8%)
MoneCon 20]
22.50
16.46
13.95
13.10
8.41
6.94
2,80
1.92
1.55
MonoCon Ours
23.31
16.36
13.73
14.90
10.28
8.70
5,38
2.89
2.83
Improvement
(+3.6%)
(-0.6%)
(-1.69)
(+13.79%)
(+-22.36)
(425.49)
(499.119)
(+50.55%
(+82.88%)
Table 3: Evaluation results on Waymo validation set.
Vehicle, APao 1 30-50m
Vehicle, APHap 1
Difticulty
Method
Overall
0-30 m
50m-2o
Overall
0-30 m
30-50 m
50 m-
L
MonoCon 20
2.30
6.66
0.67
0.02
2.29
6.62
0.66
0.02
MonoCon Ours
2.50
7.62
0.72
0.02
2.48
7.57
0.72
0.02
(IOU 0.7)
Improvement
(+8.7%)
(4144.4%)
(47.5%
(40.0%)
(48.3%
(414.4%)
(-9.1%)
(4-0.0%)
MonoCon
2.16
6.64
0.64
0.02
2.15
6.59
0.64
0.02
_
MonoCon Ours
2.34
7.59
0,70
0.02
2.33
7,54
0.69
0.02
=OUU=077
Impravement
(18335)
(1.34)
(1944%)
(10.0%
(18.4%)
(i (1.44)
(17.8)
(100055
LEVEL_1
MonoCon
10.07
27.47
3.84
0.16
9.99
27.26
3.81
0.16
ManoCon Ours
10.14
28.51
3.99
0.17
10.06
28.28
3,96
0.17
(1OU= 0.5)
Improvement
(+0.7%)
(+3.8%)
(+3.99)
(+6.3%)
(+0.77%
(+3.7%)
(+3.9%)
(+6.3%%
MonoCon
9.44
27.37
3.71
0.14
9.37
27.17
3,68
0.14
LEVEL_
MonoCon Ours
9.50
28.40
3.85
0.15
9.43
28.18
3.82
0.15
(IOU= 0.5)
Improvement
(+0.6%)
(+3.8%
(+3.8%
(+7.1%)
(+0.6%)
(+3.7%)
(+3.8%
(+7.1%)
only for Car but also for Pedestrian and Cyclist. The average performance increases for these object categories are 10.3%, 12.4%, and 53.1 %, respectively.
Waymo dataset. We extend the evaluations the Waymo dataset further demonstrate the gener- ality of our method, as shown in Tab. 3 The results indicate that our proposed method consistently enhances performance by an average of 4.6% and 4.5% in terms of mAP and mAPH metrics across all levels, IoU thresholds, and distance ranges. Significantly, the proposed loss boosts the depth discrimination not only within specific depth ranges but across all depth ranges. This underscores the effectiveness of our method not just on the KITTI dataset, but on the Waymo dataset as well.
5.2 Additional Experiments
Ablation studies. To demonstrate the effectiveness of our proposed method, we conduct the ablation studies of two components of our method: (K, B, e) "--quasi-isomettie loss Eyt and object- wise depth map loss Looj. We employ DID-M3D [27] and MonoCon (20) as baselines of a method with and without extra data. The results in Tab. 4 indicate that any method incorporating either the quasi-isometric or object-wise depth losses sees a performance improvement. Notably, the (K, B, c)- quasi-isometric loss contributes to larger performance gain than the object-wise depth loss (+2.3% vs. +1.9% for DID-M3D and +8.7% vs. +5.0% for MonoCon). The models trained with both of the proposed losses significantly surpass the performance of the baselines.
Comparison of our method with SupCR. To demonstrate the advantage of our quasi-isometric loss over existing metric learning schemes, we compare the task performance of our method with that of SupCR (Supervised Contrastive Regression) 39) SupCR is the first regression- learning method that effectively applies metric learning to regression tasks using GT labels. Similar to our quasi-isometric loss Lgi, SupCR selectively chooses the relative negative pair object features based on positive pair distance. Tab. 5 we report the AP3dIRa0 for Car; Moderate and errors between the GT and prediction of four key tasks that determine the location of the 3D bounding box: 1 E 2, (h, w. 1), 7, e, independently. The errors include the absolute difference F.,, Edim for 2, (h, w. (), respectively, the mean angular distance Ay [ for 7, and the accuracy Acc.c for object
representation
8

---

Table 4: Ablation studies on KITTI validation set.
Components
Car, APadjaw 1 Moderate
Baseline
Overall
Lagi
Lobj
Easy
Hard
23,93
16.22
13.98
0.0%
24.61 (+2.8%)
16.57 (+2.2%)
14,23 (+1.8%)
+2.3%
DID-M3D 27
24.53 (+-2.5%)
16.84 (+3.8%)
13.87 (-0.8%)
+1.8
24.77 (+3.5%
17.12 (+5.5%%
14.30 (+2.3%)
+3.8%
23.03
17.84
15.37
0.0%
27.00 (+117.2%)
18.68 (+4.77%)
16.02i +44.2%)
+8.7%
MonoCon 20
24.90 (488.1%)
18.65 (44.5%%
15.73 (4-2.3%)
+5.0%
27.90 (+21.1%)
19.43 (+8.9%)
16.92 (+-00.3%)
+ +1.5%
Table 5: Comparison of our method with SupCR on KITTI validation set.
Baseline
APaDIRao 1
Eg (m) +
Edim (m) 4
Ay (rad)
Acc.c (%)
MonoCon 20
17.84
0.019
0.025
7/711.9
93.15
MonoCon SupCR
17.55 (- 1.6%)
0.014 (-26%%)
0.031 (424.0%)
7/112.18 (+18.99%)
91.11 (-2.2%)
MonoCon Las
18.68 (-4.7%)
0.011 (-42.1%)
0.024 (-4.0%
7/208.11 (+1.0%)
93.07 (-0.1%)
classification c. The results illustrate that our method enhances 3D object detection performance, whereas SupCR diminishes it. Interestingly, both methods reduce depth errors Eg as they train the encoder to extract more depth-discriminatiee features. However, SupCR significantly increases the errors in bounding box size and angle estimation Edim, Dy by approximately 24.0% and 18.9%, respectively. It demonstrates that SupCR, which forcibly arranges the feature manifold, can negatively impact the performance of other tasks due the complex shared representations across multiple sub-tasks.
Scalability of our quasi-isometric loss with the anchor-based method and BEV paradigm. In Tab. 4 we note that our method improves performance when applied to CenterNet-based baselines (anchor-free). We further demonstrate the effectiveness of the proposed metric learning incorporated into anchor- mnchor-based methods and the bird-eye-view (BEV) paradigm: MonoDTR [5] and Im Nexe [30]. The object-wiss wise depth map loss is not applied to these baselines, since it already uses an auxiliary depth loss or inherent nature of the BEV paradigm.
As shown in Tab. [6] although the performance enhancement compared to anchor-free methods is not significant, quasi-isometric loss still shows consistent improvements across all metries for each baseline. These results suggest that the loss has broad applicability to the tasks that extract object features on image spatial coordinates given object depth labels.
Model performance with respect to the ratio of property-violated objects. The proposed quasi-isometric loss is designed arrange the feature space in accordance with object depth labels, aiming meet quasi-isometric proper- ties. To demonstrate the correlation between 3D object detection performance and the ratio of fea- tures that satisfy the quasi-isometric condition, we conduct an additional experiment. In Fig. 2 we plot the ratio of object pairs violating the properties (where the Ratio = (IP*++|P 1/IP),
property-wiolated pairs
19:75
Parformance
Ratio (%)
39.25
18.75
18.23
37.75
22-1
Loss scale
PP (2) and L is the total number of ob- jects in the dataset.) and model performance in relation to various loss scales Agi. For this exper- iment, we use the baseline model 200 trained with different loss scales [10-5 1,003, 10-2, 10-1, 0.5). We measure the AP with ToU thresholds of 0.7 for Car, Moderate on the KITTI validation dataset. The results indicate that an increase the loss scale corresponds with a decrease in the ratio of object pairs violating the inasi-c properties, leading to improvement in model performance. This
Figure 2: Loss scale (log-scaled) with respect to property-violated pairs ratio and performance.

---

Table 6: Evaluation results of anchor-based and bird-eye-viee paradigms on KITTI validation set.
Car, A Moderate
Method
Overall
Easy
Hard
MonoDTR [5]
24.57
18.45
15.37
MonoDTR Ours
26.17 (+6.5%)
19.07 (+3.4%)
15.93 (+3.6%)
+4.5%
Im Nexe 30
24.54
17.80
15.87
Im Voxe Ours
26.14 (+88.0%)
18.20 (+7.2%)
15.81 (+3.8%)
+6.3%
Table 7: Performance of KITTI validation with respect to K, B,
Components
Car, APsD) R40 Moderate
Overall
Easy
Hard
MonoCon (wlo Cob))
23.03
17.84
15.37
+0.0%
1.0
0.5
10.0
24.67 (+7.1%)
18.09 (+1.44%
15.34 (-0.2%)
+2.8%
1,5
0.5
10.0
27.00 (+ (7.1%)2%
18.68 (+4.7%)
16.02 (4 (+.4.2%)
+8.7%
2.0
0.5
10.0
23.34 (+1.33%
17.44 (-2.2%)
14.80 (-3.7%)
-1.5%
1.5
0.0
10.0
24.54 (1 6.6%%
18.01 (1 (1.00%)
15.37 (1 (.0%)
+2.5%
1.5
1.0
10.0
25.49 (0.7%)
18.70 (+44.8%)
15.82 (4-2.9%)
+6.1%
1.5
5.0
10.0
23.70 (+2.99%
18.00 (+0..%%
15.12 (+-1.6%)
+0.9%
0.5
1.0
24.71 (+7.3%%
17.84 (+0.0%%
15.43 (40..4%)
+2.6%
5.0
24.99 (+8.5%%
18.23 (+2.2%%
(+2.8%%
+4.5%
1.5
0.5
00
22.85 0.8%)
16.91 5.2%)
14.14 ( 8.0%)
-4.7%
suggests a potential relationship between the reduction in the ratio of property-violated object pairs and the enhancement of model performance.
Influence of pyperaarameters B,, in quasi-isometric loss. The choice of the tunable hyperparameter e crucial. An excessively small would sample a few objects within mini-batch, making representation learning infeasible. On the other hand, quasi-isometric properties with too large an 6 would harm the non-linearity of the feature manifold in P-space. To investigate the influence of hyperparameters on our quasi-isometric loss, we evaluate the performance of "Monocon + Ours" using KITTI [2] validation set for various hyperparameters (K, B,e). as presented Tab, 7 Although most hyperparameter combinations improved performance, there are notable exceptions, especially when = 2.0 and = Jc. We observe a decline in performance with excessively large values of K or e, which define the criteria for property violation and the non-linearity of the feature manifold.
Conclusion
In this paper, address the challenge of monocular 3D object detection in RGB images by proposing a novel metric learning scheme. Our method, which does not rely on extra parameters, modules, or data, concentrates extracting denth-discriminative features without increasing the inference time model size. By employing distance-preserving preserving function and the (K, B, e) -)-guasi-isoment successfully arrange the feature space manifold in accordance with ground-truth object depth, while preserving the non-lineen non-linearity of the natural feature manifold. Furthermore, by introducing an auxiliary head for object-wise depth estimation, we improve object depth quality without increasing inference time. Our experimental results on the KITTI and Waymo datasets illustrate consistent performance enhancements across different baselines, highlighting the effectiveness of our proposed method. As potential avenue for future work, our method could feasibly be extended to multi-camera 3D object detection scenarios and other regression tasks that involve multiple sub-tasks.
loss, we
Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00210900
10

---

References
[1] Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4))433- 459, 2010.
[2] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1333-1396 2003.
[3] Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In ICCV, pages 9287-9296, 2019.
[4] Stephen G Brick. Quasi-isometriess and ends of groups. Journal of pure and applied algebra, 86(1):23-33, 1993.
[5] Shaosheng Cao, Wei Lu, and Qiongkai Xu. Deep neural networks for learning graph representa- tions. In AAAI, volume 30, 2016.
[6] Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.
[7] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CVPR, pages 15750-15758, 2021.
[8] Yi-Nan Chen, Hang Dai, and Yong Ding. Pseudo-stereo for monocular 3d object detection in autonomous driving. In CVPR, pages 887-897, 2022.
[9] Yongjian Chen, Lei Tai, Kai Sun, and Mingyang Li. Monopair: Monocular 3d object detection using pairwise spatial relationships. In CVPR, pages 12093-12102, 2020.
[10] Mingyu Ding. Yuqi Huo, Hongwei Yi. Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. Learning depth-guided convolutions for monocular 3d object detection. In CVPRW, pages 1000-1001, 2020.
[11] David Figen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. In ICCV, pages 2650-2658, 2015.
[12] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. CVPR, 2012.
[13] Mikhael Gromov, Misha Katz, Pierre Pansu, and Stephen Semmes. Metric structures for Riemannian and non- Riemannian spaces, volume 152. Springer, 1999.
[14] Yu Hong, Hang Dai, and Yong Ding. Cross-modality knowledge distillation network for monocular 3d object detection. In ECCV, pages 87-104. Springer, 2022.
[15] Kuan-Chill Huang, Tsung-Har Han Wu, Hung-Ting Su, and Winston H Hsu. Monodtr: Monocular 2022.
3d object detection with depth- aware transformer. In CVPR, pages
[16] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NeurlPS, 33:18661- 18673, 2020.
[17] Joseph Kruskal. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29(1):1-27, 1964.
[18] Abhinav Kumar, Garrick Brazil, Enrique Corona, Armin Parchami, and Xiaoming Liu. Deviant: Depth equivariant network for monocular 3d object detection. In ECCV, pages 664 683. Springer, 2022.
[19] Qing Lian, Botao Ye, Ruijia Xu, Weilong Yao, and Tong Zhang. Exploring geometric consistency for monocular 3d object detection. In CVPR, pages 1694, June 2022.
[20] Xianpeng Liu, Nan Xue, and Tianfu Wu. Learning auxiliary monocular contexts helps monocular 3d object detection. In AAAI, volume 36, pages 1810-1818 2022.

---

[21] Zechen Liu, Zizhang Wu, and Roland Tóth. Smoke: Single-stage monocular 3d object detection via keypoint estimation. In CVPRW, pages 996-997, 2020.
[22] Yan Lu, Xinzhu Ma, Lei Yang. Tianzuu Zhang, Yating Liu, Qi Chu, Junjie Yan, and Wanli Ouyang. Geometry uncertainty projection network for monocular 3d object detection. In ICCV, pages 3111-3121, 2021.
[23] Xinzhu Ma, Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang. Delving into localization errors for monocular 3d object detection. In CVPR, pages 4721 1721-4730, 2021.
[24] Fabian Manhardt, Wadim Kehl, and Adrien Gaidon. Roi-1Od: Monocular lifting of 2d detection to 6d pose and metric shape. In CVPR, pages 2069-2078, 2019.
[25] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In CVPR, pages 7074-7082, 2017.
[26] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidat needed for monocular 3d object detection? In ICCV, pages 3142-3152, 2021.
[27] Liang Peng, Xiaopei Wu, Zheng Yang. Haifeng Liu, and Deng Cai. Did-m3d: Decoupling instance depth for monocular 3d object detection. In ECCV, pages 71-88. Springer, 2022.
[28] Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander. Categorical depth distribution network for monocular 3d object detection. In CVPR, pages 8564, 2021.
[29] Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290(5500):2323 2226,, 2000.
[30] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image voxels projection for monocular and multi-view general-purposs. 3d object detection. In WACV, pages 2397-2406, 2022.
[31] Adrian Spurr, Aneesh Dahiya, Xi Wang. Xucong Zhang, and Otmar Hilliges. Self-super 3d hand pose estimation from monocular rgb via contrastive learning. In ICCV, pages 11230-11233 12399, 2021.
[32] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaík, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, pages 2446-2454, 2020.
[33] Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 29065500):231-23225 2000.
[34] Daixin Wang, Peng Cui, and Wenwu Zhu. Structural deep network embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1234, 2016.
[35] Yan Wang. Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinherger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, pages 8445-844 8453, 2019.
[36] Yaoming Wang. Yangzhou Jiang. Jin Li, Bingbing Ni, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Teng Li. Contrastive regression for domain adaptation on gaze estimation. CVPR, pages 19376-19385. 2022.
[37] Zizhang Wu, Yunzhe Wu, Jian Pu, Xianzhi Li, and Xiaoquan Wang. Attention-based depth distillation with 3d-aware positional encoding for monocular 3d object detection. arXiv preprint arXiv:2211.16779 2022.
[38] Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese. Data-driven 3d voxel patterns for object category recognition. In CVPR, pages 1903-1911, 2015.
[39] Kaiwen 7ha, Peng Cao, Yuzhe Yang, and Dina Katabi. Supervised contrastive regression. arXiv preprint arXiv:2210.01180 2022.
12

---

[40] Renrui Zhang, Han Qiu, Tai Wang, Xuanzhuo Xu, Ziyu Guo, Yu Qiao, Peng Gao, and Hongsheng Li. Monodetr: Depth-aware transformer for monocular 3d object detection. arXiv preprint arXiv:2203. 13310, 2022.
[41] Yunpeng Zhang, Jiwen Lu, and Jie Zhou. Objects are different: Flexible monocular 3d object detection. CVPR, pages 2289-,298, 2021.
[42] Zhenyue Zhang and Hongyuan Zha. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM journal scientific computing, 26(1):313-333) 2004.
[43] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects points. arXiv preprint 17X700190,.07850, 2019.
13


Reviews:
[{"review_id": "w8kWflEruk", "reviewer": "Reviewer_Wa1k", "summary": "This paper proposes a metric learning scheme to learn depth-discriminative features for object depth prediction, which helps improve the overall task of monocular 3D object detection, without negatively impacting the performance of the other sub-tasks (e.g., object class, bounding box size) wherein. Specifically,  they employ a distance-preserving function and the proposed ($K, B, ε$)-quasi-isometric loss to arrange the feature space manifold in accordance with ground-truth object depth, while preserving the non-linearity of the natural feature manifold. They also introduce an auxiliary head (in training) for object-wise depth estimation to enhance the depth quality. Experiments on datasets KITTI and Waymo show that the proposed method can be incorporated into several 3D object detection backbones for improvement.", "questions": "- Are those hyper-parameters $K$, $B$ and $\\epsilon$ easy to find in pratice? Will different backbone architectures require different setups of $K$, $B$, and $\\epsilon$?\n- According to Eq 6, the proposed Quasi-isometric loss is implemented as the contrastive loss. \n   1) Are there any restrictions on the absolute number and the relative ratio of positive/negative samples/anchors? \n   2) Is there any direct comparison between the proposed Quasi-isometric loss with the naive contrastive loss for metric learning in the task of monocular 3D object detection?\n- It would be helpful to show some visualization results of the learned features.", "limitations": "- N/A for the limitations. \n- I suggest the authors provide the failure cases (and the corresponding explanations) of the proposed losses on monocular 3D object detection.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed $(K, B, ε)$-quasi-isometric loss is mathematically explained to help learn depth-discriminative features.\n- Experiments on KITTI and Waymo show the effectiveness of the proposed method, and the ablation studies are well-designed for verification including several backbones and comparison with other SOTA (e.g., SupCR).\n- The paper is well written, including the problem definition, the purpose, and the background. The equations and the detailed notations help to understand the method. ", "weaknesses": "See **Questions** below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** Are those hyper-parameters $K, B,$ and $\\epsilon$ easy to find in practice? Will different backbone architectures require different setups of $K, B,$ and $\\epsilon$?\n\n**A1.** Finding the *\"Optimal\"* hyper-parameters can be challenging across diverse backbones and datasets. However, our experiments spanning multiple datasets and baselines consistently used the hyperparameters: $(K=1.5, B=0.5, \\epsilon=10.0)$. These settings consistently enhanced the 3D object detection performance across various setups.\n___\n>**Q2.** According to Eq 6, the proposed Quasi-isometric loss is implemented as the contrastive loss.\n- Are there any restrictions on the absolute number and the relative ratio of positive/negative samples/anchors?\n- Is there any direct comparison between the proposed Quasi-isometric loss with the naive contrastive loss for metric learning in the task of monocular 3D object detection?\n\n**A2-1.** Our quasi-isometric loss does not have rigid constraints. Most 3D detection networks, like MonoCon [6], cap at 30 objects per image. If there are no positive/negative samples, the loss returns zero, courtesy of the NT-Xent loss design.\n\n**A2-2.** Since object depth estimation is a regression-oriented task, most conventional contrastive learning methods aren't directly applicable, with the exception of SupCR [3]. Upon reviewers' requests, we adapted the two prominent contrastive learning schemes, SimCLR [1] and SupCon [2], which were originally designed for classification tasks, to enable feature learning. SupCon is specifically tailored to enhance feature learning by harnessing the information derived from the GT depth (refer to global response for details).\n\nSame as the proposed loss $\\cal{L_{qi}}$, all contrastive losses were applied to object feature $\\rho$.\nOur results in `R-Table 1` highlight the comparison of the proposed $\\cal{L_{qi}}$ to the modified contrastive losses.\nA detailed description of each contrastive loss implementation can be found in the global response.\n\nRegarding $\\cal{L_\\text{SimCLR}}$, it underperforms the baseline substantially. This is because $\\cal{L_\\text{SimCLR}}$ focuses on extracting discriminative features of the object even when the object features share identical GT depth. When employing $\\cal{L_\\text{SupCon}}$ or $\\cal{L_\\text{SupCR}}$ as the loss function, there is a marked improvement in performance over $\\cal{L_\\text{SimCLR}}$.\nThis underscores the advantage of integrating depth label information during the training phase.\nIt is noteworthy that $\\cal{L_\\text{SimCLR}}$ is self-supervised and does not leverage depth label information.\n\nWe experimented with SupCon's approach denoted as $\\cal{L_\\text{SupCon v2}}$, where only $\\cal{L_\\text{SupCon}}$ was used to train the backbone feature space in the early stage. Later stages saw the training of task classifiers with $\\cal{L_\\text{baseline}}$ while freezing the backbone. But this setup fails to predict the 3D bounding box because the other sub-task classifiers of the model could not differentiate the pre-set feature space that was defined solely by depth.\n\nIn conclusion, the method incorporating our quasi-isometric loss $\\cal{L_{qi}}$ significantly surpasses those utilizing other contrastive losses, demonstrating an impressive margin of $+7.6$%p to $+23.6$%p.\nThis is attributed to our quasi-isometric loss, which prioritizes neighboring samples and fine-tunes the feature manifold, all the while conserving its original shape and the intrinsic non-linearity derived from various tasks.\nFor further insights, kindly refer to **A2** in response to **R2 (6BaQ)**.\n___\n>**Q3.** It would be helpful to show some visualization results of the learned features.\n\n**A3.** Following your suggestion, we included the visualization results of the feature spaces learned by our proposed method, alongside those derived from various metric learning methods in `R-Figure 1`. For clarity, the points represent projected object feature points, and their associated colors indicate depth GT values.\n___"}, {"review_id": "D7qNDqKksN", "reviewer": "Reviewer_6BaQ", "summary": "The paper proposed a new approach to monocular 3D object detection. The critical contribution of the work is the application of metric learning to improve depth estimation and an additional head with auxiliary depth prediction. The resulting approach improves 3D object detection accuracy without increasing inference time, model size, or additional data.\n\nThe proposed method utilizes a metric learning scheme that preserves the geodesic distance between depth information and the feature space. This scheme encourages the model to extract depth-discriminative features without negatively impacting other non-depth tasks (object size, type and etc.). An auxiliary head is also introduced to enhance depth estimation, adapting from [20]. This auxiliary head improves the quality of depth estimation without impacting inference time, ensuring efficient performance.\n\nThe experimental evaluations conducted on the KITTI and Waymo datasets demonstrate the effectiveness of the proposed method. The results consistently show solid improvements in performance across various monocular 3D object detection methods.\n\n------ Updated my score after rebuttal.", "questions": "- Can I ask why not use nuScenes, which seems to be more popular in monocular 3D detection in the past few years?\n- How does P correspond to the feature map h in section 3.2? Is P the feature space that the map predicts? Or is a different one going through some MLP?\n- Is there any systematic way for parameter choosing? Even just a starting set?\n- Have the authors tried other metric learning methods that didn't seem to work?", "limitations": "The parameter of the loss seems to be quite sensitive to the final performance. Would it be helpful to provide guidelines or an algorithm for parameter initialization?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. The authors did a great job describing the quasi-isometric concept, its mathematics, and how the proposed loss enforces them.\n- The paper is well-intuitive. The different asks in 3D object detection can create conflict in feature extraction, negatively impacting each task. The idea of using metric learning to provide a feature manyfold that balances the discrimination of depth and local structure for other tasks is promising. \n- The related work section is comprehensive, covering different related areas.\n- The experiment demonstrates the strength of the proposed method and generalizability on different baselines and datasets. Ablation analysis is done on the two key components to show their effectiveness and contribution.", "weaknesses": "- The paper provides two essential components contributing to the final improvement. But I don't think the second auxiliary head contains a lot of novelty compared to related works such as MonoCon. This limits the novelty of the paper to some extent.\n- It's great that the paper compared to SupCR in ablation analysis. However, there is a missing discussion from a more theoretical standpoint on why the proposed loss is better than contrastive loss. In the end, both of the losses seem to be able to maintain a certain local distance for the other tasks.\n- Part of the parameter-sensitive analysis should be moved to the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** I don't think the second auxiliary head contains a lot of novelty compared to related works such as MonoCon. This limits the novelty of the paper to some extent.\n\n**A1.** We acknowledge your concern regarding the novelty of the auxiliary head.\nWhile it may not present a significant technical novelty, it uses an object-wise depth map as a subordinate task to mitigate errors in a depth task stemming from center shifting.\nThis is because CenterNet-based models rely on the exact center point pixel, assuming objects as points.\n___\n>**Q2.** There is a missing discussion from a more theoretical standpoint on why the proposed loss is better than contrastive loss. In the end, both of the losses seem to be able to maintain a certain local distance for the other tasks.\n\n**A2.** Our quasi-isometric loss addresses the negative transfer problem in multi-task learning due to two primary reasons.\nFirst, the quasi-isometric loss inherently has a capable margin, denoted as $(K, B)$, which ensures the distinctiveness of other sub-task classifiers.\nTo illustrate, consider two objects: one from the car class and the other from the pedestrian class.\nEven though both objects are at the same depth, our loss ensures their distinguishability.\nIn contrast, other contrastive losses relying on GT depth tend to consistently aggregate feature points based solely on depth, neglecting the discriminability required for other sub-tasks.\nThis is further described in `R-Table 1`.\n\nSecond, our quasi-isometric loss benefits from the incorporation of a local distance-preserving condition.\nThis ensures a structured arrangement of the feature manifold while maintaining its intricate overall shape.\nFor instance, let's imagine the feature space being modeled by a subset of the circle manifold, as depicted in `R-Figure 2-(a)`.\nFrom a depth perspective, this manifold represents a structured feature space since the distance of all object feature pairs along the geodesic corresponds closely with depth distance.\nHowever, without the local distance-preserving condition, as shown in `R-Figure 2-(b)`, the quasi-isometric loss might erroneously infer that features $p4$ and $p5$ violate property norms.\n\nOn the other hand, integrating the local distance-preserving condition denoted by $\\epsilon$ (`R-Figure 2-(c)`) refines the quasi-isometric loss to only consider neighbor samples.\nThis approach enables a more nuanced arrangement of the feature manifold while preserving the overall shape and non-linearity of the original feature space.\n___\n>**Q3.** Parameter-sensitive analysis should be moved to the main paper.\n\n**A3.** Thank you for your constructive feedback. We will report the parameter-sensitive analysis in the revised version.\n___\n>**Q4.** Can I ask why not use nuScenes, which seems to be more popular in monocular 3D detection in the past few years?\n\n**A4.**\nMany monocular 3D object detection studies have favored evaluation on the KITTI [7] and Waymo [8], with fewer focusing on the nuScenes [4].\nHeeding your suggestion, we have now expanded our evaluation to include the nuScenes dataset, as presented in `R-Table 4`.\nThe proposed method achieved a $2.8$% improvement in MAE, and a notable $12.3$% enhancement in Car AP, resulting in an overall performance boost of $7.6$%.\nThese consistent gains across all datasets underline the robustness and generalization ability.\n___\n>**Q5.** How does P correspond to the feature map h? Is P the feature space that the map predicts? Or is a different one going through some MLP?\n\n**A5.**\nWe directly extract the object feature $\\rho$ from the feature map $\\mathbf{h}$ by referencing the spatial coordinate of GT coarse projected 3d center $(u,v)$.\n___\n>**Q6.** Is there any systematic way for parameter choosing? Even just a starting set?\n\n**A6.**\nWhile the systematic approach to parameter selection was not a focal point of the main paper, we did identify two critical observations that assist in this process:\nFirst, for datasets with a large number of objects, selecting a stricter $B$ can enhance performance.\nThis is achieved by approximately preserving the pseudo-geodesic distance in the $\\mathbf{P}$-space. (Refer to Section B in supplementary materials)\nSecond, as mentioned in the main paper, both excessively small or large values of $\\epsilon$ can hinder representation learning.\nSpecifically, a small $\\epsilon$ risks sampling an insufficient number of property-violated object pairs.\nOn the other hand, a large $\\epsilon$ can compromise the non-linearity of the feature manifold within $\\mathbf{P}$-space.\n\nWe suggest beginning with the parameter set of $(K=1.5, B=0.5, \\epsilon=10.0)$, as our experiments have shown that this configuration consistently boosts performance across multiple baselines and datasets.\nWe acknowledge that the pre-defined parameters may detract from the robustness of our method.\nHowever, future research could address this issue, possibly through methods like automated parameter search, among other potential solutions.\n___\n>**Q7.** Have the authors tried other metric learning methods that didn't seem to work?\n\n**A7.**\nIn our main paper, we excluded results from conventional metric learning methods like SimCLR [1] and SupCon [2] since they are primarily tailored for classification tasks.\nInstead, we chose to compare ours with SupCR [3], which is designed specifically for regression tasks.\nThis is because the methods [1,2] are not directly applicable to object depth estimation as a regression task.\n\nTo highlight the advantages of our approach over conventional contrastive losses, we modified two renowned contrastive learning techniques: SimCLR [1] and SupCon [2].\nSupCon is specifically tailored to enhance feature learning by harnessing the information derived from the GT depth (refer to global response for details).\n\nDue to space constraints, we kindly ask you to refer to **A2-2** of **R1 (Wa1k)** for a comprehensive analysis of the experiment.\n___"}, {"review_id": "2nrbV8dQ2Q", "reviewer": "Reviewer_nNTt", "summary": "One main challenge of monocular 3D object detection models is the lack of depth information from RGB images. The authors proposed a metric learning scheme to encourage the model to extract depth-discriminative features. Based on the presented theoretical results, the authors proposed a quasi-isometric loss and an object-wise depth map loss to supervise the model. Quantitative results on benchmark datasets supported the main arguments of the paper and sufficient ablation study experiments were conducted.", "questions": "1. What is “Ours” in supplementary Table 3? Only $\\mathcal{L}\\_{qi}$ or both $\\mathcal{L}\\_{qi}$ and $\\mathcal{L}\\_{obj}$? I assume when ablating $K, B, \\epsilon$, you should not involve $\\mathcal{L}\\_{obj}$ in the comparison?\n2. Since there is an auxiliary head for depth estimation, is the estimated depth more accurate with the quasi-isometric loss? It would be good to compare the trade-off between depth estimation and “other sub-tasks” when $\\epsilon$ changes.", "limitations": "1. The work can be improved by comparing with methods with similar ideas (see weakness 1).\n2. It is a bit unclear how different feature spaces look like under different hyper-parameter settings. It would be good if more analysis are presented (see weakness 2).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation and problem setting are explained clearly. References were provided to demonstrate why depth is key to improve current monocular 3D methods and how depth-discriminative features would help.\n2. The proposed quasi-isometric loss seems novel and effective. To prevent the depth-discriminative loss from damaging the non-linearity of the natural manifold, the authors adopted a distance-preserving condition. Despite extra hyper-parameters introduced, the proposed method is effective in general under various settings.\n3. Quantitative results on two benchmark datasets using multiple baseline models demonstrated the effectiveness of the method. Results showed that the proposed depth-discriminative is an effective approach to assist monocular 3D methods.", "weaknesses": "1. The high-level idea of the proposed approach resembles previous contrastive learning approaches [1]. I could imagine adding depth-based feature contrastive losses to the baseline loss. Would that work? What would be the advantage of the proposed approach compared to contrastive losses?\n2. The authors claimed that directly learning quasi-isometry would hurt sub-tasks. Despite such trade-offs being common in deep learning, I think this problem is a bit understudied. For instance, object features from objects that differ a lot in distance should be easily discriminated against. Would larger $K$ help with a larger $\\epsilon$? Or is this design related to hard negative sampling?\n3. Figure 2 seems interesting. However, I assume the pairs are also defined by the given $K, B, \\epsilon$, which makes it less convincing.\n\nReferences:\n1. D. Neven et al. Towards End-to-End Lane Detection: an Instance Segmentation Approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** The high-level idea of the proposed approach resembles previous contrastive learning approaches. I could imagine adding depth-based feature contrastive losses to the baseline loss. Would that work? What would be the advantage of the proposed approach compared to contrastive losses?\n\n**A1.**\nIn our main paper, we excluded results from conventional metric learning methods like SimCLR [1] and SupCon [2] since they are primarily tailored for classification tasks.\nInstead, we chose to compare ours with SupCR [3], which is designed specifically for regression tasks.\nThis is because the methods [1,2] are not directly applicable to object depth estimation as a regression task.\n\nTo highlight the advantages of our approach over conventional contrastive losses, we modified two renowned contrastive learning techniques: SimCLR [1] and SupCon [2].\nSupCon is specifically tailored to enhance feature learning by harnessing the information derived from the GT depth (refer to `R-Table 1` for details).\n\nDue to space constraints, we kindly ask you to refer to **A2-2** of **R1 (Wa1k)** for a comprehensive analysis of the experiment.\n___\n>**Q2.** The authors claimed that directly learning quasi-isometry would hurt sub-tasks. Despite such trade-offs being common in deep learning, I think this problem is a bit understudied. For instance, object features from objects that differ a lot in distance should be easily discriminated against. Would larger $K$ help with a larger $\\epsilon$? Or is this design related to hard negative sampling?\n\n**A2.** Our quasi-isometric loss addresses the negative transfer problem in multi-task learning due to two primary reasons.\nFirst, the quasi-isometric loss inherently has a capable margin, denoted as $(K, B)$, which ensures the distinctiveness of other sub-task classifiers.\nTo illustrate, consider two objects: one from the car class and the other from the pedestrian class.\nEven though both objects are at the same depth, our loss ensures their distinguishability.\nIn contrast, other contrastive losses relying on GT depth tend to consistently aggregate feature points based solely on depth, neglecting the discriminability required for other sub-tasks.\nThis is further described in `R-Table 1`.\n\nSecond, our quasi-isometric loss benefits from the incorporation of a local distance-preserving condition.\nThis ensures a structured arrangement of the feature manifold while maintaining its intricate overall shape.\nFor instance, let's imagine the feature space being modeled by a subset of the circle manifold, as depicted in `R-Figure 2-(a)`.\nFrom a depth perspective, this manifold represents a structured feature space since the distance of all object feature pairs along the geodesic corresponds closely with depth distance.\nHowever, without the local distance-preserving condition, as shown in `R-Figure 2-(b)`, the quasi-isometric loss might erroneously infer that features $p4$ and $p5$ violate property norms.\n\nOn the other hand, integrating the local distance-preserving condition denoted by $\\epsilon$ (`R-Figure 2-(c)`) refines the quasi-isometric loss to only consider neighbor samples.\nThis approach enables a more nuanced arrangement of the feature manifold while preserving the overall shape and non-linearity of the original feature space.\n\nIn summary, the pre-defined hyperparameters $(K,B,\\epsilon)$ play a crucial role in determining the strictness of the local quasi-isometric property-violated condition.\nAs you rightly pointed out, increasing the value of $K$ can potentially relax the strict condition set by a larger $\\epsilon$.\nHowever, it is essential to underscore that an excessively large $K$ cannot effectively promote depth discriminative features in objects. \nMoreover, an excessively large $\\epsilon$ also harms the non-linearity of the feature manifold.\n\nFor empirical clarity, we conducted an experiment with MonoCon [6] + $\\cal{L}_{qi}$, setting $K=5.0$ and $\\epsilon=\\infty$, which we deliberately set to excessively large values.\nThis resulted in a degraded performance, registering a car class mAP of $17.51$ (moderate), in comparison to the $17.84$ (moderate) attained by the standard MonoCon.\n___\n>**Q3.** Figure 2 seems interesting. However, I assume the pairs are also defined by the given $K, B,$ and $\\epsilon$, which makes it less convincing.\n\n**A3.** We agree that the pre-defined hyper-parameters $(K, B, \\epsilon)$ diminish the elegance of the proposed quasi-isometric loss. In the near future, we plan to consider solutions like automated parameter searches to address the issue.\n___\n>**Q4.** What is “Ours” in supplementary Table 3? Only $\\cal{L_{qi}}$ or both $\\cal{L_{qi}}$ and $\\cal{L_{obj}}$? I assume when ablating $K, B,$ and $\\epsilon$, you should not involve $\\cal{L_{obj}}$ in the comparison?\n\n**A4.**\nWe appreciate your attention to detail. In `S-Table 3`, \"Ours\" includes both $\\cal{L_{qi}}$ and $\\cal{L_{obj}}$. We revisited the experiments for `S-Table 3` and updated `R-Table 2` accordingly.\nDue to space constraints, we provided only partial hyperparameter setups but will include more details on additional hyperparameter setups in the revised version.\nNotably, the trends observed in the experimental results remain consistent.\nWe identified a degradation in performance with excessively large values of $K$ or $\\epsilon$.\n___\n>**Q5.** Since there is an auxiliary head for depth estimation, is the estimated depth more accurate with the quasi-isometric loss? It would be good to compare the trade-off between depth estimation and “other sub-tasks” when $\\epsilon$ changes.\n\n**A5.** In `M-Table 4`, the ablation study for $\\cal{L_{obj}}$ and $\\cal{L_{qi}}$ shows that the marginal performance gains closely match the cumulative ones, highlighting improved depth estimation. In response to your feedback, we have conducted additional experiments adjusting epsilon and detailed the findings in `R-Table 5` to examine its impact on depth estimation relative to other tasks.\n___"}, {"review_id": "xMXBulDFy0", "reviewer": "Reviewer_tr2v", "summary": "This paper introduces a novel metric learning scheme for extracting more depth-discriminative features in the monocular 3D object detection task. A distance-preserving function is adopted to build the relation between feature space and the ground-truth object depth. The authors propose a quasi-isometric loss to adjust the distances among object descriptors. Furthermore, an auxiliary head for object-wise depth estimation is used during training, enhancing depth quality while maintaining the inference time. The experiments show that the proposed method can improve the performance significantly of various baselines on KITTI and Waymo datasets.", "questions": "See weaknesses.", "limitations": "The authors do not address the limitations of their work. This method may be limited by the number of objects.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper introduces a metric learning method to improve the discrimination of object descriptors according to their depth information. The scheme is simple yet effective, which maintains the geodesic distance of depth information in feature space. The quasi-isometry under the local distance-preserving condition mitigates the negative damage to the non-linearity of the natural manifold. It minimizes the influence on other tasks.\n\n- The quasi-isometric loss is designed to arrange condition-violated samples. The samples far from the quasi-isometric distance are pulled closer. The samples too close in feature space are pushed away. The loss focuses on hard samples which avoids many unnecessary computations while ensuring the isometric objective.  \n\n- The proposed method is plug and play. It can easily integrate into various monocular 3D object detection baselines and brings significant improvement.", "weaknesses": "- I am curious about the efficiency of the proposed method during training. The calculations of relative distances between objects in both original and feature space would be enormous. Besides, the complexity of calculating the distance matrix grows with the square of the number of objects. Therefore, the proposed method may be not suitable for the scenario with a large number of objects during training. The paper does not discuss this problem.\n \n\n- The object-wise depth prediction head is trained for estimating more accurate depth, avoiding significant errors that existed in the center depth prediction head. Why not directly use the prediction of object-wise depth head for a more accurate depth estimation during inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** I am curious about the efficiency of the proposed method during training. The calculations of relative distances between objects in both original and feature space would be enormous. Besides, the complexity of calculating the distance matrix grows with the square of the number of objects. Therefore, the proposed method may be not suitable for the scenario with a large number of objects during training. The paper does not discuss this problem.\n\n**A1.**\nAs you pointed out, the time/memory complexity of calculating the distance matrix increases quadratically with the number of objects.\nNevertheless, we enhanced the efficiency of our quasi-isometric method at the implementation level by loop unrolling as detailed in the algorithm table of supplementary materials.\nInstead of computing each distance pair-by-pair, we substituted these operations as matrix operations, which are GPU-friendly operations.\n\nFollowing your request, we empirically measure the training cost of our quasi-isometric loss.\nWe compared the training time of one epoch between the 'MonoCon' [6] and 'MonoCon+Ours' using a single NVIDIA Titan RTX. \nWe simulated the most challenging scenario by setting the maximum number of property-violated object pairs to 30 per image, given that MonoCon can infer up to 30 objects.\n\nMethod|Avg. training time (sample/ms)|Single epoch time (s)\n---|---|---\nMonoCon|$84.9$|$315$\nMonoCon+Ours (worst case)|$87.1~(+2.59$%$)$|$324~(+2.86$%$)$\n\nThe results were as follows: 'MonoCon' required 315 seconds for one epoch, whereas 'MonoCon+Ours' took 324 seconds.\nThis represents a mere $2.86$% increase in training time, which we believe has a negligible impact on overall computational efficiency.\n___\n>**Q2.** The object-wise depth prediction head is trained for estimating more accurate depth, avoiding significant errors that existed in the center depth prediction head. Why not directly use the prediction of object-wise depth head for a more accurate depth estimation during inference?\n\n**A2**. When utilizing object-wise depth maps for depth prediction, issues such as occlusion arise.\nFor instance, if the estimated center point is obscured by the 2D bounding box of a foreground car, the system might mistakenly extract the depth information of the foreground vehicle.\nThis can lead to a decline in performance.\nExperimental results confirm this: 'MonoCon+Ours' achieved a car class mAP (mean average precision) of $19.43$ (moderate), whereas 'MonoCon+Ours using object-wise depth map for prediction depth' only managed a car class mAP of $15.17$ (moderate).\n___"}, {"review_id": "7JcUZDs0Vk", "reviewer": "Reviewer_eeNQ", "summary": "This work focuses on the monocular 3D object detection task. As many works indicated, depth estimation is the bottleneck of this task, and the authors propose the apply metric learning to improve the accuracy of the depth estimation sub-task. The proposed metric-learning-based loss encourages the model to extract depth-discriminative features regardless of the visual attributes without increasing inference time and model size. Extensive experiments on the KITTI3D and Waymo Open datasets demonstrate the effectiveness of the proposed loss fuction.", "questions": "See weaknesses. ", "limitations": "See weaknesses.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. I like the idea that applying metric learning to improve the accuracy of depth estimation. Theoretically, this method has a good generalization ability and is easy to be embedded in other baseline models.\n2. Extensive experiments and good performance. The authors conduct lots of experiments on the KITTI3D and the large-scale Waymo Open datasets. Besides, they test their metric learning-based loss function on several baseline models, and the improvements across several settings demonstrate the effectiveness of their proposed model.\n3. The main idea is easy to follow and this paper is well-organized. This paper has many mathematical details that scholars in this field may not be familiar with, but it still presents them clearly and systematically.\n\n", "weaknesses": "1. In lines 68-69, the authors claim their proposed work is 'the first approach that applies metric learning to monocular 3D object detection.'  In fact, there is another work [1] that discussed how to apply metric learning in monocular 3D object detection with a focus on dimension estimation. This claim should be modified or removed.\n\n2. Although this work provides lots of experiments to show the effectiveness of their method, lots of them are based on the same foundation model, i.e. CenterNet (and one of them is based on the transformer-based pipeline). However, there are still some other popular detection pipelines such as the BEV paradigm. It will be better to validate the effectiveness of more pipelines.\n\n3. This work conducts experiments on the large-scale Waymo dataset, which is great. However, I still want to see the performance on the nuScenes dataset, because the nuScenes benchmark is dominated by another detection pipeline and adopts different evaluation metrics. Evaluating on this dataset can further show the effectiveness and generalization ability of the proposed method.\n\n\nOverall, this paper proposes a novel and effective metric-learning-based loss function. I tend to accept this work and I can further improve my rating if the authors can show the generalization ability of this work in more settings.\n\n[1] Dimension Embeddings for Monocular 3D Object Detection, Zhang et al., CVPR'22", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** In lines 68-69, the authors claim their proposed work is 'the first approach that applies metric learning to monocular 3D object detection.' In fact, there is another work that discussed how to apply metric learning in monocular 3D object detection with a focus on dimension estimation. This claim should be modified or removed.\n\n**A1.** We were not previously aware of the cited work. We appreciate your bringing it to our attention and will make the necessary revisions accordingly.\n___\n>**Q2.** Although this work provides lots of experiments to show the effectiveness of their method, lots of them are based on the same foundation model, *i.e.* CenterNet (and one of them is based on the transformer-based pipeline). However, there are still some other popular detection pipelines such as the BEV paradigm. It will be better to validate the effectiveness of more pipelines.\n\n**A2.** We greatly appreciate your insightful feedback.\nFollowing your request, we conducted an additional experiment on ImVoxelNet [9] that adopts the BEV paradigm using the 3D voxel feature.\nThis baseline reformulated the outdoor 3D object detection as 2D detection in the BEV plane. We thus extracted the object feature from the projected BEV feature prior to the head that consists of two parallel 2D convolution layers.\n\nResults, as shown in `R-Table 3`, show that while the performance gain isn't as pronounced as with CenterNet-based methods, the proposed $\\cal{L_{qi}}$ consistently improves performance across all metrics, resulting in an overall performance boost of $+6.6$%.\nThese findings, as highlighted in the main paper, underscores the broad utility of $\\cal{L_{qi}}$ in 3D object detection, provided object depth labels are available.\n___\n>**Q3.** This work conducts experiments on the large-scale Waymo dataset, which is great. However, I still want to see the performance on the nuScenes dataset, because the nuScenes benchmark is dominated by another detection pipeline and adopts different evaluation metrics. Evaluating this dataset can further show the effectiveness and generalization ability of the proposed method.\n\n**A3.** In response to your suggestion, we have now expanded our evaluation to include the nuScenes dataset [4], as presented in `R-Table 4`.\nThe proposed method achieved a $+2.8$% improvement in MAE, and a notable $+12.3$% enhancement in Car AP, resulting in an overall performance boost of $+7.6$%.\nThese consistent gains across all datasets underline the robustness and generalization ability.\n___"}]

Author rebuttal:
We thank all reviewers for their constructive comments.
For your convenience, please download and assess the **attached PDF**.
To simplify cross-referencing, figures, and tables in the main paper, supplementary materials, and rebuttal paper are denoted as `M-[Table X/Figure X]`, `S-[Table X/Figure X]`, and `R-[Table X/Figure X]`, respectively.

Below, we detail additional experiments and visualizations conducted per reviewers' requests:
___
>In response to **R1 (Wa1k)**, we included visualization results of the MonoCon [6] feature space learned by both our proposed quasi-isometric loss method and other metric learning methods using PCA in `R-Figure 1`. The points and their corresponding colors represent the projected object feature points and ground-truth (GT) depths, respectively.
___
>Addressing the inquiries from **R2 (6BaQ)** and **R3 (nNTt)**, we introduced `R-Figure 2` to explain the theoretical standpoint of the proposed quasi-isometric loss.
In this figure, $z/p$ and $\Delta \text{z}/\Delta \text{p}$ represent the depth/object-feature and Minkowski distance of the depth/object-feature pair, respectively.
Specifically, $\Delta \text{z} = d_1(z_1, z_i), \Delta \text{p} = d_2(p_1, p_i)$, with $i = \{2,3,4,5\}$ and $d_1(\cdot, \cdot), d_2(\cdot, \cdot)$ as the Minkowski distance metrics.
In `R-Figure 2-(b)` and `R-Figure 2-(c)`, the white areas represent property-satisfied zones where properties are met, while the gray areas indicate ignore zones where representation learning between object pairs is hindered due to the constraints of the $\epsilon$-ball.
___
>Based on the feedback from **R1 (Wa1k)**, **R2 (6BaQ)** and **R3 (nNTt)**, we added `R-Table 1` to compare the proposed quasi-isometric loss with other previous contrastive losses.
Since existing contrastive losses, excluding SupCR [3], are designed for classification, we adjusted each contrastive loss for object depth estimation, which is a regression problem, and a sub-task of monocular 3d object detection:
>>$\cal{L_{\text{SimCLR}}}$ [1]: We select all possible object feature pairs as negative pairs, excluding identical entity object features.
>
>>$\cal{L_{\text{SupCon}}}$ [2]: We segmented the continuous depth interval into 5-meter bins, treating them as classes, and then trained the backbone and classifier with the original SupCon loss.
>
>>$\cal{L_{\text{SupCon v2}}}$: We initially trained the backbone feature space with only $\cal{L_\text{SupCon}}$, and subsequently trained the task classifiers with $\cal{L_\text{baseline}}$ loss while freezing the backbone.
>
>>$\cal{L_{\text{SupCR}}}$: We use the same experimental setup as in the main paper.
>
>For a fair comparison, we adopted NT-Xent for loss form and cosine similarity as the metric. 
All existing contrastive learning methods make the positive pairs via two-view augmentation techniques, specifically horizontal flipping and color jittering.
___
>In response to **R3 (nNTt)**, we revised and presented the performance comparison based on hyperparameters $K, B,$ and $\epsilon$ in `R-Table 2`.
___
>Responding to **R5 (eeNQ)**, we applied our quasi-isometric loss to a BEV-based 3D object detection method, ImVoxelNet [9].
The evaluation results on the KITTI [7] **$\textit{validation}$** set can be found in `R-Table 3`.
While the implementation details for ImVoxelNet remain consistent with those mentioned in the paper, we made a modification in the batch size due to GPU memory constraints. Specifically, the batch size of ImVoxelNet was adjusted to 4, which is half of its original batch size.
___
>Per the requests from **R2 (6BaQ)** and **R5 (eeNQ)**, we performed additional experiments on the nuScenes dataset [4] with the proposed method. The results are in `R-Table 4`.
We used a split [5] comprising 28,130 train and 6,019 validation images from the front camera.
The split is convertible to the KITTI format using the script [10], facilitating comparison between MonoCon and MonoCon+Ours on the nuScenes dataset.
Metrics applied include the Mean Absolute Error (MAE) of the depth of the bounding boxes and Average Precision [4] (AP), defined by matching through thresholding the 2D center distance (d) on the ground plane, rather than using Intersection over Union (IoU).
___
>Following **R3 (nNTt)**'s suggestion, we added the performance metrics of object depth estimation with varying epsilon values and the trade-off for other sub-tasks in `R-Table 5`.
___
$\mathbf{References}$

[1] *Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." ICML 2020.*

[2] *Khosla, Prannay, et al. "Supervised contrastive learning." NIPS 2020.*

[3] *Zha, Kaiwen, et al. "Supervised Contrastive Regression." arXiv (2022).*

[4] *Caesar, Holger, et al. "nuscenes: A multimodal dataset for autonomous driving." CVPR 2020.*

[5] *Shi, Xuepeng, et al. "Geometry-based distance decomposition for monocular 3d object detection." ICCV 2021.*

[6] *Liu, Xianpeng, Nan Xue, and Tianfu Wu. "Learning auxiliary monocular contexts helps monocular 3D object detection." AAAI 2022.*

[7] *Geiger, Andreas, Philip Lenz, and Raquel Urtasun. "Are we ready for autonomous driving? the kitti vision benchmark suite." CVPR 2012.*

[8] *Sun, Pei, et al. "Scalability in perception for autonomous driving: Waymo open dataset." CVPR 2020.*

[9] *Rukhovich, Danila, Anna Vorontsova, and Anton Konushin. "Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection." WACV 2022.*

[10] https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/scripts/export_kitti.py

Meta-review: This paper proposes a metric learning scheme to learn depth-discriminative features for monocular 3D object detection. It designs a new loss consisting of a quasi-isometric loss and an object-wise depth map loss to train the model using the ground-truth object depth information. Furthermore, an auxiliary head for object-wise depth estimation is used during training. All reviewers acknowledged the novelty, effectiveness, clarity, and good performance of the proposed method and recommended the acceptance of the paper. The authors' rebuttal responds well to the reviewers' concerns; the results of the additional experiments are excellent, and several reviewers raised the final rating score. For these reasons, the paper is well worth acceptance to this conference. It is strongly recommended that the final version of the paper include the discussion and experimental results provided in the rebuttal.
Decision: Accept (poster)