Title: Depth-discriminative Metric Learning for Monocular 3D Object Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: Monocular 3D object detection

Autonomous driving

Recognition

Regression

Metric learning
Abstract: Monocular 3D object detection poses a significant challenge due to the lack of depth information in RGB images. Many existing methods strive to enhance the object depth estimation performance by allocating additional parameters for object depth estimation, utilizing extra modules or data. In contrast, we introduce a novel metric learning scheme that encourages the model to extract depth-discriminative features regardless of the visual attributes without increasing inference time and model size. Our method employs the distance-preserving function to organize the feature space manifold in relation to ground-truth object depth. The proposed $(K,B,\epsilon)$-quasi-isometric loss leverages predetermined pairwise distance restriction as guidance for adjusting the distance among object descriptors without disrupting the non-linearity of the natural feature manifold. Moreover, we introduce an auxiliary head for object-wise depth estimation, which enhances depth quality while maintaining the inference time. The broad applicability of our method is demonstrated through experiments that show improvements in overall performance when integrated into various baselines. The results show that our method consistently improves the performance of various baselines by 23.51\% and 5.78\% on average across KITTI and Waymo, respectively.

Depth-discriminative Metric Learning for Monocular 3D Object Detection

Abstract

Monocular 3D object detection poses a significant challenge due to the lack of depth information in RGB images. Many existing methods strive to enhance the object depth estimation performance by allocating additional parameters for object depth estimation, utilizing extra modules or data. In contrast, we introduce a novel metric learning scheme that encourages the model to extract depth-discriminative features regardless of the visual attributes without increasing inference time and model size. Our method employs the distance-preserving function to organize the feature space manifold in relation to ground-truth object depth. The proposed ( K,B,ϵ ) -quasiisometric loss leverages predetermined pairwise distance restriction as guidance for adjusting the distance among object descriptors without disrupting the non-linearity of the natural feature manifold. Moreover, we introduce an auxiliary head for object-wise depth estimation, which enhances depth quality while maintaining the inference time. The broad applicability of our method is demonstrated through experiments that show improvements in overall performance when integrated into various baselines. The results show that our method consistently improves the performance of various baselines by 25.27% and 4.54% on average across KITTI and Waymo, respectively.

1 Introduction

Monocular 3D object detections [20, 22, 23, 27] has gained prominence as a cost-effective and easily deployable solution, playing a critical role in autonomous driving and robotic navigation systems. Typically, the frameworks consist of a feature extractor and lightweight multi-head architectures that predict the projected centers, depths, bounding box sizes, and heading directions of multiple objects from a single RGB image [18, 20, 22, 23, 26, 27, 41]. Among these tasks, object depth estimation, inferring the distance from a monocular camera to the center of an object, is the most challenging subtask due to depth ambiguity. Previous work [23] has highlighted this issue in ablation studies, revealing that substituting object depth predictions with ground-truth (GT) values significantly improves overall performance, while replacing other sub-tasks, such as heading direction and 3D size, does not notably enhance performance. Furthermore, several prior works [3, 10, 18, 22, 27, 28, 37, 41] have attempted to improve object depth estimation quality by adding additional modules or introducing new formulations. However, despite the subsequent increase in inference time and model size, the improvement in performance remains limited.

These observations suggest that monocular 3D object detection heavily relies on object depth quality; nevertheless, conventional methods yield unsatisfactory results due to the extraction of lessdiscriminative features for object depth inference. One reason is that the extracted feature involves visual attributes, such as the color, size, and heading direction of objects, resulting in the object

depth head receiving feature with limited depth discernment. To improve object depth performance, the network should be capable of extracting purely depth-discriminative features that involve essential geometric information, irrespective of the visual attribute. A feasible method for extracting the depth-discriminative features involves using deep metric learning schemes such as contrastive learning [6, 7, 39] or representation learning [5, 34].

However, most existing deep metric learning schemes [6, 7, 16, 39] rely on aggressive two-view augmentation ( i.e. Affine transform) to train the distance or similarity metric. This aggressive data augmentation method is hardly leveraged in current monocular 3D object detection frameworks due to the violation of geometric constraints, where horizontal flip and color distortion are the only two methods used in this field for a long time [19]. One alternative way is to learn a regression-aware representation by contrasting samples against each other based on their target distance using GT labels [39]. However, forcibly arranging the manifold of the feature space using this depth distance metric may negatively impact the performance of the other tasks, because the feature extractor for monocular 3D object detection inevitably produces complex shared representations across multiple sub-tasks. Note that the experiment of negative impact is conducted in Sec. 5.2.

To address these issues, we propose a ( K,B,ϵ ) -quasi-isometric loss, a new metric learning approach that encourages the network to extract depth-discriminative features using object depth labels. Inspired by several manifold learning schemes [29, 33], we locally preserve the neighborhood distances to maintain the natural non-linear manifold of feature space in order to mitigate the negative transfer effect on other sub-tasks. Our approach utilizes the quasi-isometric properties providing a relaxed condition for the distance metric between the depth and feature metric spaces. This enables the model to arrange the feature space with respect to the object depth labels while maintaining the performance of other sub-tasks. Moreover, we introduce an auxiliary head for object-wise depth estimation to further improve object depth estimation. The head component is removed after the training process, ensuring that the inference time remains unaffected and does not experience an increase. Experimental results indicate that the proposed method consistently outperforms state-of-the-art baselines across a variety of 3D object detection datasets. Our method has been shown to be compatible with various monocular 3D object detection frameworks [15, 20, 22, 23, 27], demonstrating its broad applicability without compromising inference time or increasing model size. The effectiveness of each proposed module is further underscored through comprehensive ablation studies.

Our contributions can be summarized as follows:

We propose a simple yet effective metric learning scheme that preserves the geodesic distance of depth information to feature space.

We present an auxiliary head for object-wise depth estimation, which enhances the depth quality without impacting the inference time, maintaining efficient performance.

Our method significantly enhances the performance of various monocular 3D object detection methods without increasing inference time and model size.

2 Related work

Monocular 3D object detection. Monocular 3D object detection can be broadly categorized into two types. The first type [3, 18, 20, 22, 23, 25, 37, 41] predicts the localization of objects of interest around an ego vehicle using only RGB images, annotations, and camera calibrations. Most of them are based on CenterNet [43]. They are divided into several sub-tasks, with a primary focus on estimating accurate object depth. The works [22] and [41] propose the formulations of object depth estimation. MonoCon [20] leverages the abundant contexts in conventional 3D bounding box annotations to integrate various auxiliary tasks. The second category leverages additional data, such as pre-trained models [8, 10, 15, 27, 28, 35] or CAD models [24, 38], to compensate for the lack of 3D information in monocular images. Some prior works utilize pre-trained depth estimators to address the lack of depth information, either by converting the monocular setups to LiDAR/Stereo environments [8, 35] or by allocating more parameters to the depth estimation task [10, 15, 40]. Alternatively, certain methods employ completed depth maps from pre-trained depth completion models [27] or pre-trained LiDAR-based detectors [8, 14] to supervise their models. Notably, DID-M3D [27] divides object depth into visual and attribute depth using completed depth maps to address the ambiguity of the object depth.

Manifold geometry preservation. Deep learning networks, consisting of continuous and differentiable (precisely, almost everywhere differentiable) layers, ensure smooth mapping of networks f : R m → R n . This guarantees that neighborhoods in R m will be mapped into neighborhoods in the embedding space R n with some amount of 'stretching' and vice versa. Manifold learning has been studied to preserve most or all of the essential information by minimizing these stretches. Many works [1, 2, 17, 29, 42] aim to create meaningful representations when embedding highdimensional data into lower dimensions. The traditional linear embedding algorithms commonly use the matrix decomposition to preserve the original variance[1] or pairwise distance [17]. More recent algorithms [2, 29, 33, 42] construct the neighborhood graphs by preprocessing the dataset using ϵ -ball or k -nearest neighbors for each point, embedding the data while preserving the neighbor samples' distance. In particular, Isomap [33] estimates the shortest path in the neighborhood graph between every pair of data points and then employs the Euclidean Multidimensional Scaling (MDS) algorithm [17] to embed the points in d dimensions with minimum distance distortion all at once. These non-linear embedding algorithms can account for the non-linear nature of the manifold by preserving the distance between neighborhood samples.

Metric learning. Metric learning is one of the machine learning techniques that aims to learn an effective distance metric between data points by using training data. To achieve this, methods such as [6, 7, 16] attempt to minimize the distance between samples from the same class while maximizing the distance between samples from different classes. These methods demonstrate improved performance for self-supervised learning by learning in an end-to-end fashion, including two-view augmentation or depending on class labels. These deep metric learning methods typically target classification tasks because the positive/negative pairs are defined as belonging to the same or different classes. More recent works [31, 36, 39] extensively apply metric learning in the context of regression. These works either directly use the target distance to contrast samples against each other [36, 39] or leverage representation learning in a semi-supervised learning scenario [31].

3 Method

Our final goal is to boost the object depth performance by extracting the depth-discriminative features without increasing inference time. Similar to common deep metric learning schemes [6, 7, 39], we propose a loss term that preserves a meaningful low-dimensional data structure in the feature space.

3.1 Preliminary

Metric space. A metric space is a mathematical concept that characterizes a set of points and a function that measures the distance between any two points in the set. Formally, a metric space can be defined as a pair ( M,d ) , where M represents a set and d is a distance function on M . The distance function d must satisfies the following axioms [13] for any three points x, y, z ∈ M :

Non-negativity: d ( x, y ) ≥ 0 and d ( x, y ) = 0 if and only if x = y .

Symmetry: d ( x, y ) = d ( y, x ) for all x, y ∈ M .

Triangle inequality: d ( x, y ) ≤ d ( x, z ) + d ( z, y ) for all x, y, z ∈ M .

Quasi-isometry. A quasi-isometry is a function between two metric spaces that preserves distances up to a constant factor, even though it may locally distort angles and distances. Let Q be a function from one metric space ( M 1 , d 1 ) to another metric space ( M 2 , d 2 ) . Q is considered a quasi-isometry from ( M 1 , d 1 ) to ( M 2 , d 2 ) if there exist constants K ≥ 1 , B ≥ 0 , and ϵ ≥ 0 such that both of the following properties hold:

∀ x 1 , x 2 ∈ M 1 : 1 K · d 1 ( x 1 , x 2 ) -B ≤ d 2 ( Q ( x 1 ) , Q ( x 2 )) ≤ K · d 1 ( x 1 , x 2 ) + B .

∀ z ∈ M 2 : ∃ x ∈ M 1 s.t. d 2 ( z, Q ( x )) ≤ ϵ .

Quasi-isometry does not necessarily require continuity [4]. This property is advantageous because other distance-preserving transformations may not possess it, and most datasets are finite. Therefore, we use these conditions as constraints to ensure that the feature space retains the geometrical information of the depth metric space.

3.2 Problem Definition

The task of monocular 3D object detection aims to predict both the object class c and the 3D bounding box b for multiple objects within an image I . Recently, CenterNet [43] has become the best practice for monocular 3D object detection, with subsequent works [18, 20, 22, 23, 27, 41] adopting its pipeline. These methods decompose the bounding box b = [ X,Y,Z,h,w,l,γ ] estimation problem into separate estimations of the coarse projected 3D center ( u, v ) , the depth of object center z , the center offset ( δu, δv ) , the 3D size dimensions ( h, w, l ) , and heading direction γ ( i.e. , the yaw angle). The 3D object center [ X,Y,Z ] is computed by back-projecting the projected center with the corresponding depth, given the intrinsic matrix of the camera K , as follows:

The networks consist of feature extractor F θ and task-specific heads G ϕ t that produce the feature maps h and the per-pixel output maps ˜ o t , where t ∈ T = { t c , t u , t v , t δu , t δv , t z , t h , t w , t l , t γ } , respectively, as follows:

where H,W represent the spatial resolution of the image, and C, H ′ , W ′ denote the channel and spatial resolution of the feature maps, which is downsampled from the image resolution. The final object-wise results o T are extracted from the non-learnable function H , given the intermediate per-pixel outputs ˜ o T = { t c ,t u ,t v ,t δu ,t δv ,t z ,t h ,t w ,t l ,t γ } , as follows:

where the number of detected objects N is not fixed and can be zero if no object is detected.

3.3 Methodology

In this paper, our primary focus is on training the feature extractor F θ with the objective of extracting features that enhance the discriminability of object depth. At the same time, we aim to preserve the discriminability of other sub-tasks, such as the projected 3D center, bounding box size, and heading directions. To achieve this, we propose a metric learning method that encourages the network to extract depth-discriminative features by leveraging object depth labels.

Given M training images I i , where i ∈ I = { 1 , ..., M } , and N i GTobjects in the image I i , we extract object descriptors ρ ( i,j ) , where j ∈ J = { 1 , ..., N i } , from feature maps h i using GT coarse projected 3D center u ( i,j ) , v ( i,j ) and obtain the corresponding GT object depths z ( i,j ) . We build a set of object descriptors P = ⋃ { i ∈ I, j ∈ J } ρ ( i,j ) and a set of corresponding object depth Z = ⋃ { i ∈ I, j ∈ J } z ( i,j ) . We then define two metric spaces ( Z , d 1 ) and ( P , d 2 ) , where d 1 and d 2 represent the Minkowski distance in Euclidean space ( i.e. , L1 distance). The finite sets Z and P consist of L elements, corresponding to objects in the dataset ( | Z | = | P | = L = ∑ i ∈ I N i ). Consequently, we can establish a one-to-one function Q between these two metric spaces as follow:

Our goal is for the function Q to enforce a quasi-isometric between Z -space and P -space by using the properties of quasi-isometry in Sec. 3.1. This can encourage the network to extract depthdiscriminative features in P -space by utilizing object depth labels in Z -space. However, enforcing a quasi-isometric between the low-dimensional Z -space and the high-dimensional P -space can damage the non-linearity of the natural manifold, potentially causing the negative transfer to other sub-tasks. Therefore, we adopt the local distance-preserving condition of the non-linear embedding methods such as Isomap [33] and LLE [29]. The revised version of the quasi-isometry condition is as follows:

(i) U = { z ∈ Z | z ∈ B } ,

(ii) ∀ z 1 ∈ Z , s.t. ∀ z 2 ∈ U z 1 : · d 1 ( z 1 , z 2 ) -B ≤ d 2 ( Q ( z 1 ) , Q ( z 2 )) ≤ K · d 1 ( z 1 , z 2 ) + B ,

(iii) ∀ ρ ∈ P : ∃ z ∈ Z , s.t. d 2 ( ρ, Q ( z )) ≤ ϵ

P-space

P-space (local)

):e-ball B,e(∈= 2)

8.3m

8.3m

Feature manifold

7.8m

e

10.1m

7.8m

10.1m

:object data point

O:property-violated

object pair

(need to be close by)

13.1m

13.1m

O:property-violated

RC

g2

RC

object pair

Z-space (immutable space)

(need tobe far away)

property-satisfied

O:(or non-neighbor)

RC

objectpair

R

IR

where B ´ z,ϵ is a ball with radius ϵ centered around ´ z . By applying the quasi-isometric properties solely to neighboring samples within the ϵ -ball, the non-linearity of the feature manifold is preserved, and the shortest curve distance between any two arbitrary samples on the manifold ( a.k.a. geodesic) in P -space is maintained instead of the Minkowski distance. By ensuring these three conditions (i), (ii), (iii) are met for a sufficiently small K ≥ 1 and B ≥ 0 , we can encourage object features ρ l to be depth-discriminative using their corresponding object depth z l . To achieve this, function Q must satisfy these conditions, which involves training the parameter θ of the feature extractor F .

(K, B, ϵ )-Quasi-isometric loss. We propose a quasi-isometric loss term that enforces the quasiisometric between the depth and feature metric spaces while preserving the distance among the neighbor data points as illustrated in Fig. 1. In particular, the proposed loss term is designed to arrange P -space samples that do not satisfy these conditions. To ensure efficient training within the constraints of limited GPU memory, we use the samples within a mini-batch instead of using the entire input dataset as done in previous works [2, 29, 33]. We selectively choose the neighbor objects in a mini-batch with respect to their corresponding depth labels, where neighboring object depths are within U z .

Suppose that objects in mini-batch images I b correspond to the object depths Z b and object features P b on Z -space and P -space, respectively. We can then identify the object feature pair sets P + b , P -b that violate the revised property (ii) by transposing it as follow:

where ( z 1 , z 2 ) is the corresponding depth pair of the object features ( ρ 1 , ρ 2 ) . For an object feature pair with distance d 2 ( ρ 1 , ρ 2 ) in P -space, the distance is larger for P + b in Eq. 5a or smaller for P -b in Eq. 5b than the corresponding depth pair distance d 1 ( z 1 , z 2 ) with a factor of K and the additive constant B . To ensure these property-violating object feature pairs P + b , P -b satisfy the property (ii), we propose the ( K,B,ϵ ) -quasi-isometry loss, which modifies the Normalized Temperature-Scaled Cross-Entropy loss (NT-Xent loss) from [6] as follows:

Algorithm 1 ( K,B,ϵ ) -Quasi-isometric loss

where τ is the temperature term, and ∥· , ·∥ p is p -norm, and K ≥ 1 , B ≥ 0 denote the pre-defined hyperparameters that determine the hardness of quasi-isometric property. This loss aims to make the property-violated object features comply with the quasi-isometric property by assuming P + / -b as positive/negative anchors, respectively. The similarity metrics S + / -are always positive values in (0 , 1] that imply a negative distance gap between the distance metric on Z -space and P -space. These adjustments are necessary for proper alignment between the two spaces, noting that P b is derived from F θ ( I b ) .

Object-wise depth map loss. According to Eq. 1, the estimated object depth ˆ z and coarse projected 3D center (ˆ u, ˆ v ) jointly determine the location of the object 3D center ( ˆ X, ˆ Y , ˆ Z ) . However, many existing methods [20, 21, 23, 27] train the depth loss on the local region exactly on the GT center ( u, v ) , which can lead to significant errors in object depth estimation even with slightly inaccurate object center (ˆ u, ˆ v ) during inference ( i.e. , center shifting in the image plane by a few pixels) [23].

To mitigate this issue, we train our network with an additional auxiliary head for object-wise depth estimation using the structure of [20]. Rather than solely providing depth supervision to an object center ( u, v ) , the auxiliary head is trained with depth supervision over the entire bounding box of the object. We create the foreground object-wise depth map D ∈ R H ′ × W ′ following the policy of [40]. Then, we define the object-wise depth loss by adopting the Laplacian aleatoric uncertainty loss [9], the same as the object depth estimation task, only for the foreground regions:

where z p , ˆ z p and ˆ σ p are the corresponding GT object depth, prediction, and uncertainty of each pixel, respectively. Note that the additional task head is removed after training, so the proposed method does not increase the inference time.

Total loss. The total loss L total combines the loss used in 3D object detection baselines L baseline , the quasi-isometric loss L qi , and the object-wise depth map loss L obj as follows:

where λ qi and λ obj represent the balancing weights for the quasi-isometric loss and the object-wise depth map loss, respectively. These values are set to 0.5 and 1. This total loss is applied to each baseline in our experiments, which we denote as '[Baseline] + Ours', as discussed further in Sec. 5.

Additionally, to simplify the implementation of our proposed ( K,B,ϵ ) -quasi-isometric loss, we detail the loss function in Alg.1.

4 Experiments

Datasets. The KITTI dataset [12] consists of 7,481 training images and 7,518 test images for official KITTI 3D object detection evaluation and contains three categories: Car , Pedestrian , and Cyclist . For additional experiments, we follow [23], which splits the training images into 3,712 and 3,769 images for training and validation sets, respectively. The Waymo dataset [32] is a recently released dataset comprising 798 training sequences and 202 validation sequences, with four categories: Vehicles , Pedestrians , Cyclists , and Signs . We use the split reported in [28], including 52,386 training and 39,848 validation images, to evaluate performance on the Waymo dataset.

Evaluation Metrics. We adhere to the protocol reported in [12] and [28] for KITTI and Waymo datasets, respectively. The KITTI 3D object detection performance is evaluated by the average precision of 3D bounding boxes ( AP 3 D | R 40 ) with IoU thresholds of 0.7 for Car and 0.5 for Pedestrian and Cyclist . The evaluation is split into three levels of difficulty: Easy, Moderate, and Hard, based on the 2D bounding box height, occlusion level, and truncation. In contrast, the evaluation metrics of the Waymo dataset are based on 3D IoU with mean average precision (3D mAP) and mean average precision weighted by heading (3D mAPH). Each object is divided into one or two levels and evaluated at three distances: [0, 30), [30, 50), and [50, ∞ ) meters.

Implementation details. First, we incorporate our method into four different baselines, including CenterNet-based frameworks [20, 22, 23, 27] for the KITTI dataset. For the Waymo dataset, we choose [20], as it performs best on the KITTI dataset among our baselines. We primarily follow the experimental setup details ( i.e. , epochs, optimizer) of each paper [20, 22, 23, 27] for a fair comparison. We denote the plugged versions of baselines as '[Baseline] + Ours'. As an exception, we set all batch sizes to 16 for the KITTI and Waymo benchmarks. We retrain all baselines on the KITTI/Waymo validation set with ' Car ' and ' Vehicle ' classes, respectively. For the ( K,B,ϵ ) -quasi-isometric loss, we use K = 1 . 5 , B = 0 . 5 , ϵ = 10 . 0 , d 1 ( · , · ) = |· , ·| , d 2 ( · , · ) = ∥· , ·∥ 2 for all experiments. We provide more detailed experimental setups for each baseline in the supplementary materials.

5 Experimental Results

5.1 Evaluation Results on KITTI and Waymo Datasets

KITTI dataset. We demonstrate the effectiveness of our method on both the KITTI validation and test datasets. As shown in Tab. 1, our method can be widely applied to various baselines and significantly enhances performance by a considerable margin. Particularly, the improvement is more pronounced in models that do not utilize extra data, surpassing models that do (13.1% vs. 3.8%). We surmise that the model trained with additional LiDAR data learns more discriminative features regarding depths than the method without extra data. Notably, the performance of 'MonoCon + Ours' exceeds that of models trained with additional LiDAR data. This result illustrates that our proposed metric learning method empowers MonoCon to extract depth-discriminative features without the need for extra data. Similar trends are observable in the evaluation with the KITTI test datasets, as shown in Tab. 2. The experiments demonstrate that our proposed method enhances depth discrimination not

only for Car but also for Pedestrian and Cyclist . The average performance increases for these object categories are 10.3%, 12.4%, and 53.1%, respectively.

Waymo dataset. We extend the evaluations to the Waymo dataset to further demonstrate the generality of our method, as shown in Tab. 3. The results indicate that our proposed method consistently enhances performance by an average of 4.6% and 4.5% in terms of mAP and mAPH metrics across all levels, IoU thresholds, and distance ranges. Significantly, the proposed loss boosts the depth discrimination not only within specific depth ranges but across all depth ranges. This underscores the effectiveness of our method not just on the KITTI dataset, but on the Waymo dataset as well.

5.2 Additional Experiments

Ablation studies. To demonstrate the effectiveness of our proposed method, we conduct the ablation studies of two components of our method: ( K,B,ϵ ) -quasi-isometric loss L qi and objectwise depth map loss L obj . We employ DID-M3D [27] and MonoCon [20] as baselines of a method with and without extra data. The results in Tab. 4 indicate that any method incorporating either the quasi-isometric or object-wise depth losses sees a performance improvement. Notably, the ( K,B,ϵ ) -quasi-isometric loss contributes to a larger performance gain than the object-wise depth loss (+2.3% vs. +1.9% for DID-M3D and +8.7% vs. +5.0% for MonoCon). The models trained with both of the proposed losses significantly surpass the performance of the baselines.

Comparison of our method with SupCR. To demonstrate the advantage of our quasi-isometric loss over existing metric learning schemes, we compare the task performance of our method with that of SupCR (Supervised Contrastive Regression) [39]. SupCR is the first regression-aware representation learning method that effectively applies metric learning to regression tasks using GT labels. Similar to our quasi-isometric loss L qi , SupCR selectively chooses the relative negative pair object features based on positive pair distance. In Tab. 5, we report the AP 3 D | R 40 for Car, Moderate and errors between the GT and prediction of four key tasks that determine the location of the 3D bounding box: t ∈ z, ( h, w, l ) , γ, c , independently. The errors include the absolute difference E z , E dim for z, ( h, w, l ) , respectively, the mean angular distance ∆ γ [11] for γ , and the accuracy Acc. c for object

classification c . The results illustrate that our method enhances 3D object detection performance, whereas SupCR diminishes it. Interestingly, both methods reduce depth errors E z as they train the encoder to extract more depth-discriminative features. However, SupCR significantly increases the errors in bounding box size and angle estimation E dim , ∆ γ by approximately 24.0% and 18.9%, respectively. It demonstrates that SupCR, which forcibly arranges the feature manifold, can negatively impact the performance of other tasks due to the complex shared representations across multiple sub-tasks.

Scalability of our quasi-isometric loss with the anchor-based method and BEV paradigm.

In Tab. 1-4, we note that our method improves performance when applied to CenterNet-based baselines (anchor-free). We further demonstrate the effectiveness of the proposed metric learning incorporated into anchor-based methods and the bird-eye-view (BEV) paradigm: MonoDTR [15] and ImVoxelNet [30]. The object-wise depth map loss is not applied to these baselines, since it already uses an auxiliary depth loss or inherent nature of the BEV paradigm.

As shown in Tab. 6, although the performance enhancement compared to anchor-free methods is not as significant, our quasi-isometric loss still shows consistent improvements across all metrics for each baseline. These results suggest that the loss has broad applicability to the tasks that extract object features on image spatial coordinates given object depth labels.

Model performance with respect to the ratio of property-violated objects. The proposed quasi-isometric loss is designed to arrange the feature space in accordance with object depth labels, aiming to meet quasi-isometric properties. To demonstrate the correlation between 3D object detection performance and the ratio of features that satisfy the quasi-isometric condition, we conduct an additional experiment. In Fig. 2, we plot the ratio of object pairs violating the properties (where the Ratio = ( | P + | + | P -| ) / | P | ,

| P | = ( L 2 ) , and L is the total number of objects in the dataset.) and model performance in relation to various loss scales λ qi . For this exper-

20.00

1.2

property-violatedpairs

19.75

Performance

10

19.50

8

19.25

Ratio

19.00

|R40|IoU>(

6

18.75

18.50

18.25

0

7

2

18.00

17.75

10-4

10-3

102

10-1

Loss scale

iment, we use the baseline model [20] trained with different loss scales [10 -5 , 10 -3 , 10 -2 , 10 -1 , 0 . 5] . We measure the AP 3 D with IoU thresholds of 0.7 for Car, Moderate on the KITTI validation dataset. The results indicate that an increase in the loss scale corresponds with a decrease in the ratio of object pairs violating the quasi-isometric properties, leading to an improvement in model performance. This

suggests a potential relationship between the reduction in the ratio of property-violated object pairs and the enhancement of model performance.

Influence of hyperparameters ( K,B,ϵ ) in quasi-isometric loss. The choice of the tunable hyperparameter ϵ is crucial. An excessively small ϵ would sample a few objects within a mini-batch, making representation learning infeasible. On the other hand, quasi-isometric properties with too large an ϵ would harm the non-linearity of the feature manifold in P -space. To investigate the influence of hyperparameters on our quasi-isometric loss, we evaluate the performance of 'Monocon + Ours' using KITTI [12] validation set for various hyperparameters ( K,B,ϵ ) , as presented in Tab. 7. Although most hyperparameter combinations improved performance, there are notable exceptions, especially when K = 2 . 0 and ϵ = ∞ . We observe a decline in performance with excessively large values of K or ϵ , which define the criteria for property violation and the non-linearity of the feature manifold.

6 Conclusion

In this paper, we address the challenge of monocular 3D object detection in RGB images by proposing a novel metric learning scheme. Our method, which does not rely on extra parameters, modules, or data, concentrates on extracting depth-discriminative features without increasing the inference time or model size. By employing a distance-preserving function and the ( K,B,ϵ ) -quasi-isometric loss, we successfully arrange the feature space manifold in accordance with ground-truth object depth, while preserving the non-linearity of the natural feature manifold. Furthermore, by introducing an auxiliary head for object-wise depth estimation, we improve object depth quality without increasing inference time. Our experimental results on the KITTI and Waymo datasets illustrate consistent performance enhancements across different baselines, highlighting the effectiveness of our proposed method. As a potential avenue for future work, our method could feasibly be extended to multi-camera 3D object detection scenarios and other regression tasks that involve multiple sub-tasks.

Acknowledgments and Disclosure of Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00210908).

References

Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics , 2(4):433-459, 2010.

Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation , 15(6):1373-1396, 2003.

Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In ICCV , pages 9287-9296, 2019.

Stephen G Brick. Quasi-isometries and ends of groups. Journal of pure and applied algebra , 86(1):23-33, 1993.

Shaosheng Cao, Wei Lu, and Qiongkai Xu. Deep neural networks for learning graph representations. In AAAI , volume 30, 2016.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , pages 1597-1607. PMLR, 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR , pages 15750-15758, 2021.

Yi-Nan Chen, Hang Dai, and Yong Ding. Pseudo-stereo for monocular 3d object detection in autonomous driving. In CVPR , pages 887-897, 2022.

Yongjian Chen, Lei Tai, Kai Sun, and Mingyang Li. Monopair: Monocular 3d object detection using pairwise spatial relationships. In CVPR , pages 12093-12102, 2020.

Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. Learning depth-guided convolutions for monocular 3d object detection. In CVPRW , pages 1000-1001, 2020.

David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In ICCV , pages 2650-2658, 2015.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR , 2012.

Mikhael Gromov, Misha Katz, Pierre Pansu, and Stephen Semmes. Metric structures for Riemannian and non-Riemannian spaces , volume 152. Springer, 1999.

Yu Hong, Hang Dai, and Yong Ding. Cross-modality knowledge distillation network for monocular 3d object detection. In ECCV , pages 87-104. Springer, 2022.

Kuan-Chih Huang, Tsung-Han Wu, Hung-Ting Su, and Winston H Hsu. Monodtr: Monocular 3d object detection with depth-aware transformer. In CVPR , pages 4012-4021, 2022.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NeurIPS , 33:1866118673, 2020.

Joseph B Kruskal. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika , 29(1):1-27, 1964.

Abhinav Kumar, Garrick Brazil, Enrique Corona, Armin Parchami, and Xiaoming Liu. Deviant: Depth equivariant network for monocular 3d object detection. In ECCV , pages 664-683. Springer, 2022.

Qing Lian, Botao Ye, Ruijia Xu, Weilong Yao, and Tong Zhang. Exploring geometric consistency for monocular 3d object detection. In CVPR , pages 1685-1694, June 2022.

Xianpeng Liu, Nan Xue, and Tianfu Wu. Learning auxiliary monocular contexts helps monocular 3d object detection. In AAAI , volume 36, pages 1810-1818, 2022.

Zechen Liu, Zizhang Wu, and Roland Tóth. Smoke: Single-stage monocular 3d object detection via keypoint estimation. In CVPRW , pages 996-997, 2020.

Yan Lu, Xinzhu Ma, Lei Yang, Tianzhu Zhang, Yating Liu, Qi Chu, Junjie Yan, and Wanli Ouyang. Geometry uncertainty projection network for monocular 3d object detection. In ICCV , pages 3111-3121, 2021.

Xinzhu Ma, Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang. Delving into localization errors for monocular 3d object detection. In CVPR , pages 4721-4730, 2021.

Fabian Manhardt, Wadim Kehl, and Adrien Gaidon. Roi-10d: Monocular lifting of 2d detection to 6d pose and metric shape. In CVPR , pages 2069-2078, 2019.

Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In CVPR , pages 7074-7082, 2017.

Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In ICCV , pages 3142-3152, 2021.

Liang Peng, Xiaopei Wu, Zheng Yang, Haifeng Liu, and Deng Cai. Did-m3d: Decoupling instance depth for monocular 3d object detection. In ECCV , pages 71-88. Springer, 2022.

Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander. Categorical depth distribution network for monocular 3d object detection. In CVPR , pages 8555-8564, 2021.

Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science , 290(5500):2323-2326, 2000.

Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. In WACV , pages 2397-2406, 2022.

Adrian Spurr, Aneesh Dahiya, Xi Wang, Xucong Zhang, and Otmar Hilliges. Self-supervised 3d hand pose estimation from monocular rgb via contrastive learning. In ICCV , pages 11230-11239, 2021.

Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR , pages 2446-2454, 2020.

Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science , 290(5500):2319-2323, 2000.

Daixin Wang, Peng Cui, and Wenwu Zhu. Structural deep network embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , pages 1225-1234, 2016.

Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR , pages 8445-8453, 2019.

Yaoming Wang, Yangzhou Jiang, Jin Li, Bingbing Ni, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Teng Li. Contrastive regression for domain adaptation on gaze estimation. In CVPR , pages 19376-19385, 2022.

Zizhang Wu, Yunzhe Wu, Jian Pu, Xianzhi Li, and Xiaoquan Wang. Attention-based depth distillation with 3d-aware positional encoding for monocular 3d object detection. arXiv preprint arXiv:2211.16779 , 2022.

Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese. Data-driven 3d voxel patterns for object category recognition. In CVPR , pages 1903-1911, 2015.

Kaiwen Zha, Peng Cao, Yuzhe Yang, and Dina Katabi. Supervised contrastive regression. arXiv preprint arXiv:2210.01189 , 2022.

Renrui Zhang, Han Qiu, Tai Wang, Xuanzhuo Xu, Ziyu Guo, Yu Qiao, Peng Gao, and Hongsheng Li. Monodetr: Depth-aware transformer for monocular 3d object detection. arXiv preprint arXiv:2203.13310 , 2022.

Yunpeng Zhang, Jiwen Lu, and Jie Zhou. Objects are different: Flexible monocular 3d object detection. In CVPR , pages 3289-3298, 2021.

Zhenyue Zhang and Hongyuan Zha. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM journal on scientific computing , 26(1):313-338, 2004.

Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint arXiv:1904.07850 , 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the monocular 3D object detection task. As many works indicated, depth estimation is the bottleneck of this task, and the authors propose the apply metric learning to improve the accuracy of the depth estimation sub-task. The proposed metric-learning-based loss encourages the model to extract depth-discriminative features regardless of the visual attributes without increasing inference time and model size. Extensive experiments on the KITTI3D and Waymo Open datasets demonstrate the effectiveness of the proposed loss fuction.\n\nSTRENGTHS:\n1. I like the idea that applying metric learning to improve the accuracy of depth estimation. Theoretically, this method has a good generalization ability and is easy to be embedded in other baseline models.\n2. Extensive experiments and good performance. The authors conduct lots of experiments on the KITTI3D and the large-scale Waymo Open datasets. Besides, they test their metric learning-based loss function on several baseline models, and the improvements across several settings demonstrate the effectiveness of their proposed model.\n3. The main idea is easy to follow and this paper is well-organized. This paper has many mathematical details that scholars in this field may not be familiar with, but it still presents them clearly and systematically.\n\nWEAKNESSES:\n1. In lines 68-69, the authors claim their proposed work is 'the first approach that applies metric learning to monocular 3D object detection.'  In fact, there is another work [1] that discussed how to apply metric learning in monocular 3D object detection with a focus on dimension estimation. This claim should be modified or removed.\n\n2. Although this work provides lots of experiments to show the effectiveness of their method, lots of them are based on the same foundation model, i.e. CenterNet (and one of them is based on the transformer-based pipeline). However, there are still some other popular detection pipelines such as the BEV paradigm. It will be better to validate the effectiveness of more pipelines.\n\n3. This work conducts experiments on the large-scale Waymo dataset, which is great. However, I still want to see the performance on the nuScenes dataset, because the nuScenes benchmark is dominated by another detection pipeline and adopts different evaluation metrics. Evaluating on this dataset can further show the effectiveness and generalization ability of the proposed method.\n\n\nOverall, this paper proposes a novel and effective metric-learning-based loss function. I tend to accept this work and I can further improve my rating if the authors can show the generalization ability of this work in more settings.\n\n[1] Dimension Embeddings for Monocular 3D Object Detection, Zhang et al., CVPR'22\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel metric learning scheme for extracting more depth-discriminative features in the monocular 3D object detection task. A distance-preserving function is adopted to build the relation between feature space and the ground-truth object depth. The authors propose a quasi-isometric loss to adjust the distances among object descriptors. Furthermore, an auxiliary head for object-wise depth estimation is used during training, enhancing depth quality while maintaining the inference time. The experiments show that the proposed method can improve the performance significantly of various baselines on KITTI and Waymo datasets.\n\nSTRENGTHS:\n- This paper introduces a metric learning method to improve the discrimination of object descriptors according to their depth information. The scheme is simple yet effective, which maintains the geodesic distance of depth information in feature space. The quasi-isometry under the local distance-preserving condition mitigates the negative damage to the non-linearity of the natural manifold. It minimizes the influence on other tasks.\n\n- The quasi-isometric loss is designed to arrange condition-violated samples. The samples far from the quasi-isometric distance are pulled closer. The samples too close in feature space are pushed away. The loss focuses on hard samples which avoids many unnecessary computations while ensuring the isometric objective.  \n\n- The proposed method is plug and play. It can easily integrate into various monocular 3D object detection baselines and brings significant improvement.\n\nWEAKNESSES:\n- I am curious about the efficiency of the proposed method during training. The calculations of relative distances between objects in both original and feature space would be enormous. Besides, the complexity of calculating the distance matrix grows with the square of the number of objects. Therefore, the proposed method may be not suitable for the scenario with a large number of objects during training. The paper does not discuss this problem.\n \n\n- The object-wise depth prediction head is trained for estimating more accurate depth, avoiding significant errors that existed in the center depth prediction head. Why not directly use the prediction of object-wise depth head for a more accurate depth estimation during inference?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors do not address the limitations of their work. This method may be limited by the number of objects.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOne main challenge of monocular 3D object detection models is the lack of depth information from RGB images. The authors proposed a metric learning scheme to encourage the model to extract depth-discriminative features. Based on the presented theoretical results, the authors proposed a quasi-isometric loss and an object-wise depth map loss to supervise the model. Quantitative results on benchmark datasets supported the main arguments of the paper and sufficient ablation study experiments were conducted.\n\nSTRENGTHS:\n1. The motivation and problem setting are explained clearly. References were provided to demonstrate why depth is key to improve current monocular 3D methods and how depth-discriminative features would help.\n2. The proposed quasi-isometric loss seems novel and effective. To prevent the depth-discriminative loss from damaging the non-linearity of the natural manifold, the authors adopted a distance-preserving condition. Despite extra hyper-parameters introduced, the proposed method is effective in general under various settings.\n3. Quantitative results on two benchmark datasets using multiple baseline models demonstrated the effectiveness of the method. Results showed that the proposed depth-discriminative is an effective approach to assist monocular 3D methods.\n\nWEAKNESSES:\n1. The high-level idea of the proposed approach resembles previous contrastive learning approaches [1]. I could imagine adding depth-based feature contrastive losses to the baseline loss. Would that work? What would be the advantage of the proposed approach compared to contrastive losses?\n2. The authors claimed that directly learning quasi-isometry would hurt sub-tasks. Despite such trade-offs being common in deep learning, I think this problem is a bit understudied. For instance, object features from objects that differ a lot in distance should be easily discriminated against. Would larger $K$ help with a larger $\\epsilon$? Or is this design related to hard negative sampling?\n3. Figure 2 seems interesting. However, I assume the pairs are also defined by the given $K, B, \\epsilon$, which makes it less convincing.\n\nReferences:\n1. D. Neven et al. Towards End-to-End Lane Detection: an Instance Segmentation Approach.\n\nQUESTIONS:\n1. What is “Ours” in supplementary Table 3? Only $\\mathcal{L}\\_{qi}$ or both $\\mathcal{L}\\_{qi}$ and $\\mathcal{L}\\_{obj}$? I assume when ablating $K, B, \\epsilon$, you should not involve $\\mathcal{L}\\_{obj}$ in the comparison?\n2. Since there is an auxiliary head for depth estimation, is the estimated depth more accurate with the quasi-isometric loss? It would be good to compare the trade-off between depth estimation and “other sub-tasks” when $\\epsilon$ changes.\n\nLIMITATIONS:\n1. The work can be improved by comparing with methods with similar ideas (see weakness 1).\n2. It is a bit unclear how different feature spaces look like under different hyper-parameter settings. It would be good if more analysis are presented (see weakness 2).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a new approach to monocular 3D object detection. The critical contribution of the work is the application of metric learning to improve depth estimation and an additional head with auxiliary depth prediction. The resulting approach improves 3D object detection accuracy without increasing inference time, model size, or additional data.\n\nThe proposed method utilizes a metric learning scheme that preserves the geodesic distance between depth information and the feature space. This scheme encourages the model to extract depth-discriminative features without negatively impacting other non-depth tasks (object size, type and etc.). An auxiliary head is also introduced to enhance depth estimation, adapting from [20]. This auxiliary head improves the quality of depth estimation without impacting inference time, ensuring efficient performance.\n\nThe experimental evaluations conducted on the KITTI and Waymo datasets demonstrate the effectiveness of the proposed method. The results consistently show solid improvements in performance across various monocular 3D object detection methods.\n\n------ Updated my score after rebuttal.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow. The authors did a great job describing the quasi-isometric concept, its mathematics, and how the proposed loss enforces them.\n- The paper is well-intuitive. The different asks in 3D object detection can create conflict in feature extraction, negatively impacting each task. The idea of using metric learning to provide a feature manyfold that balances the discrimination of depth and local structure for other tasks is promising. \n- The related work section is comprehensive, covering different related areas.\n- The experiment demonstrates the strength of the proposed method and generalizability on different baselines and datasets. Ablation analysis is done on the two key components to show their effectiveness and contribution.\n\nWEAKNESSES:\n- The paper provides two essential components contributing to the final improvement. But I don't think the second auxiliary head contains a lot of novelty compared to related works such as MonoCon. This limits the novelty of the paper to some extent.\n- It's great that the paper compared to SupCR in ablation analysis. However, there is a missing discussion from a more theoretical standpoint on why the proposed loss is better than contrastive loss. In the end, both of the losses seem to be able to maintain a certain local distance for the other tasks.\n- Part of the parameter-sensitive analysis should be moved to the main paper.\n\nQUESTIONS:\n- Can I ask why not use nuScenes, which seems to be more popular in monocular 3D detection in the past few years?\n- How does P correspond to the feature map h in section 3.2? Is P the feature space that the map predicts? Or is a different one going through some MLP?\n- Is there any systematic way for parameter choosing? Even just a starting set?\n- Have the authors tried other metric learning methods that didn't seem to work?\n\nLIMITATIONS:\nThe parameter of the loss seems to be quite sensitive to the final performance. Would it be helpful to provide guidelines or an algorithm for parameter initialization?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a metric learning scheme to learn depth-discriminative features for object depth prediction, which helps improve the overall task of monocular 3D object detection, without negatively impacting the performance of the other sub-tasks (e.g., object class, bounding box size) wherein. Specifically,  they employ a distance-preserving function and the proposed ($K, B, ε$)-quasi-isometric loss to arrange the feature space manifold in accordance with ground-truth object depth, while preserving the non-linearity of the natural feature manifold. They also introduce an auxiliary head (in training) for object-wise depth estimation to enhance the depth quality. Experiments on datasets KITTI and Waymo show that the proposed method can be incorporated into several 3D object detection backbones for improvement.\n\nSTRENGTHS:\n- The proposed $(K, B, ε)$-quasi-isometric loss is mathematically explained to help learn depth-discriminative features.\n- Experiments on KITTI and Waymo show the effectiveness of the proposed method, and the ablation studies are well-designed for verification including several backbones and comparison with other SOTA (e.g., SupCR).\n- The paper is well written, including the problem definition, the purpose, and the background. The equations and the detailed notations help to understand the method.\n\nWEAKNESSES:\nSee **Questions** below.\n\nQUESTIONS:\n- Are those hyper-parameters $K$, $B$ and $\\epsilon$ easy to find in pratice? Will different backbone architectures require different setups of $K$, $B$, and $\\epsilon$?\n- According to Eq 6, the proposed Quasi-isometric loss is implemented as the contrastive loss. \n   1) Are there any restrictions on the absolute number and the relative ratio of positive/negative samples/anchors? \n   2) Is there any direct comparison between the proposed Quasi-isometric loss with the naive contrastive loss for metric learning in the task of monocular 3D object detection?\n- It would be helpful to show some visualization results of the learned features.\n\nLIMITATIONS:\n- N/A for the limitations. \n- I suggest the authors provide the failure cases (and the corresponding explanations) of the proposed losses on monocular 3D object detection.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}