OpenBox: Annotate Any Bounding Boxes in 3D
Authors: In-Jae Lee
Mungyeom Kim
Kwonyoung Ryu
Pierre Musacchio
Jaesik Park
Venue: NeurIPS 2025 spotlight ()
Keywords: 3D Object Detection
Autonomous Driving
Automatic Annotation
Open-Vocabulary
Primary area: applications
TL;DR: Automatic Annotation for 3D Object Detection using 2D visual foundation model.

Abstract: Unsupervised and open-vocabulary 3D object detection has recently gained attention, particularly in autonomous driving, where reducing annotation costs and recognizing unseen objects are critical for both safety and scalability. However, most existing approaches uniformly annotate 3D bounding boxes, ignore objects’ physical states, and require multiple self-training iterations for annotation refinement, resulting in suboptimal quality and substantial computational overhead. To address these challenges, we propose OpenBox, a two-stage automatic annotation pipeline that leverages a 2D vision foundation model. In the first stage, OpenBox associates instance-level cues from 2D images processed by a vision foundation model with the corresponding 3D point clouds via context-aware refinement. In the
second stage, it categorizes instances by rigidity and motion state, then generates adaptive bounding boxes with class-specific size statistics. As a result, OpenBox produces high-quality 3D bounding box annotations without requiring self-training.
Experiments on the Waymo Open Dataset (WOD), the Lyft Level 5 Perception dataset, and the nuScenes dataset demonstrate improved accuracy and efficiency over baselines.

OpenBox: Annotate Any Bounding Boxes in 3D
In-Jae Lee
Mungyeom Kim!
Kwonyoung Ryu²
Pierre Musacchio'
Jaesik Park
Seoul National University
POSTECHH
We introduce OpenBox, which utilizes 2D vision foundation model to annotate 3D bounding boxes automatically. It annotates instances of vehicles, pedestrians, and cyclists. We demonstrate it with Waymo Open Dataset [33]. Best viewed in color and zoomed in.
Figure 1:
Abstract
Unsupervised and open- vona-uloablar 3D object detection have recently gained at- tention, particularly in autonomous driving, where reducing annotation costs and recognizing unseen objects are critical for both safety and scalability. However, most existing approaches uniformly annotate 3D bounding boxes, ignoring ob- jects' physical states, and require multiple self-training iterations for annotation refinement, resulting in suboptimal quality and substantial computational over- propose OpenBox, two-stage automatic
head. To address these challenges, annotation pipeline that leverages 2D vision foundation model. In the first stage, OpenBox associates instance-level cues from 2D images processed by vision foundation model with the corresponding 3D point clouds via cross-modal instance alignment. In the second stage, it categorizes instances by rigidity and motion state, then generates adaptive bounding boxes with class-specific size statistics. As result, OpenBox produces high-quality 3D bounding box annotations with- out requiring self-training. Experiments on the Waymo Open Dataset (WOD), the Lyft Level 5 Perception dataset, and the nuScenes dataset demonstrate im- proved accuracy and efficiency over baselines. Our project page is available at: https://liverd9222gihubb.b www.ggibiiooooonBoo...
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
3D object detection has become increasingly important across a wide range of applications, including autonomous driving [16, 21, 23, 42], robotics [29, 41], and virtual reality [15, 37J. In autonomous driving, it provides essential inputs for motion prediction that, in turn, inform path planning and vehicle control. As result, the accuracy of 3D object detection is directly tied to the overall safety and reliability of the system. While recent advances in deep learning have significantly improved detection performance, most existing frameworks [16, 21, 23, 42] remain limited to a fixed set of object categories and are heavily reliant on large-scale, human- annotated datasets. This closed-set assumption becomes particularly problematic in open-world autonomous driving scenarios. In such settings, the system must be able to detect a wide range of object types, including rare or previously unseen instances.
detection enables models to recognize arbitrary categories based on semantic descriptions, thereby overcoming the limitations of fixed label space. In the 2D image domain, open-vocabulary perception has been accelerated by the availability of large-scale image-text paired datasets and the emergence of vision foundation models. These models demonstrate strong generalization capabilities across tasks such as classification [30, 34], detection [22, 24, 48], and segmentation [14, 19,311.
Integrating
Despite the advances above, creating large-scale annotated 3D datasets remains a major bottleneck. Unlike 2D images. LiDAR point clouds provide precise geometric structure but lack rich semantic context, making them difficult to align with text-based supervision and challenging to annotate manually. To address these limitations, several unsupervised methods [40, 43, 45, 46] have been proposed. These typically follow a pipeline in which ground points are removed from raw LiDAR scans, spatial clustering is applied to extract object instances, and scene flow [1, 25, 26] or persistence point score (PP Score) [40, 43, 46] is used to identify motion states. The resulting 3D bounding boxes are then refined through multiple rounds of self-training [43, 45, 46] or sampling strategies [46]. However, these methods generally do not consider physical properties of instances for box annotation, which leads to low-quality boxes and remains computationally expensive due to their iterative refinement. More recently, several works [18, 26, 46] incorporate image semantics to assist automatic annotation. Nevertheless, [46] fuses modality-specific 3D bounding boxes the output level without geometric alignment, and [18] does not fully leverage visual cues to improve 3D annotation quality.
This paper proposes a two-stage pipeline, OpenBox, that automatically annotates 3D bounding box for arbitrary classes. Our approach leverages high-quality instance-level information from 2D vision foundation models (c.g., Grounding DINO [22]. SAM2 [31]) as supervisory signals, thereby reducing the cost and time of manual annotation. In the first stage (Cross-modal Instance Alignment), 2D instance-lever information is unprojected onto the 3D point clouds. To address noisy or incomplete instance point clouds caused by the vision foundation model, apply a context-aware refinement step to enhance the quality of instance-level points. Subsequently, the refined instances are categorized into three physical types: static rigid, dynamic rigid, and deformable. Based category-specile object size statistics, we generate 3D bounding boxes for each category. Specifically, we construct mesh for static rigid objects using the Signed Distance Function (SDF) [36] and filter out noise points through majority voting. We then further refine the bounding box via 3D-2D IoU alignment and visibility. We conduct experiments on the WOD [33], Lyft [12], and nuScenes [2] datasets. Qualitative results on real-world data show that our method produces high-quality and robust 3D annotations, as illustrated in Fig. 1.
Our contributions are summarized as follows:
We propose OpenBox, a novel automatic annotation pipeline that requires only synchronized ego poses, images, and LiDAR point clouds, without self-training.
To improve point clouds quality, we introduce a two refinement process: context-aware refinement and surface-aware noise filtering based on the SDE We also generate bounding boxes adaptively based on the physical types of instances.
Training with OpenBox- OpenBox-generated annotations achieves 70.49% AP for the vehicle class of the WOD [33] at 0.5 IoU. On the Lyft dataset [12], OpenBox outperforms the state-of-the-art approach by +19.94% AP when directly compared to human annotation boxes.

---

Input Images and LiDAR scans
Cross-modal Instance Alignment
Adaptive 3D Bbox Generation
3D Bbox Annotation
See 3.1
See 3.2
Figure 2: Pipeline Overview of OpenBox. With time-synchronized, unlabeled images and LiDAR seans, cross-modal instance alignment (Sec 3.1) associates 2D instance cues with corresponding point clouds. Adaptive 3D bounding box generation (Sec 3.2) independently chooses the most suitable fitting strategy for each instance, yielding high-quality 3D bounding boxes.
Related Work
2.1
Open-vocabulary 3D Object Defection
The rapid progress of 2D vision foundation models [11, 20, 22, 30, 34] has spurred active research in open-vocabulary 3D object detection. Most existing methods [8, 26, 44] focus on accurately aligning 2D visual cues with 3D spatial information. UP-VL [26] enhances the MI-UP [25] auto-labeling pipeline by incorporating OpenSeg [11] to generate semantically aligned amodal 3D bounding boxes for open- voccbuocayllrry transfer. Additionally, it introduces loss function that facilitates 2D-3D mapping, allowing the model to learn point-level features guided by distillation loss. Find and Propagate [8] generates frustum-shaper 3D proposals using 2D open-vocabulary detectors [20, 24], followed by multi-view alignment and density-bart based filtering to improve the detection of distant objects. OpenSight [44] lifts 2D bounding boxes obtained from Grounding DINO [22] into 3D space to enable generic object perception followed by semantic interpretation. Whereas prior work redesign 3D object detectors for the open-vocabulary setting, focus annotating the dataset to allow open-vocabulary 3D detection.
2.2 Unsupervised 3D Object Detection
LiDAR-basee Most unsupervised 3D object detection methods [1, 25, 40, 43, 45] solely rely on LiDAR point clouds to perform automatic annotation. Common pipelines first estimate motion states using PP score [40, 43] or scene flow [1, 25], and then perform ground removal followed by point cloud clustering [3. 7]. Except for [40], which utilizes class-wise size statistics, these limitations primarily stem from the lack of semantic information inherent in LiDAR, especially when compared RGB images. As a result, they generally train and evaluate models in class-agnostic manner. Moreover, CPD [40] incurs additional computational overhead by jointly using dense prototypes (CProto) and Nown-sampled point clouds, resulting in significantly longer training time.
Multi-modal based. Several approaches [18, 26, 46) use 2D vision foundation models to incorporate image information. LiSe [46] integrates 3D bounding boxes obtained from the LiDAR branch (via [43]) and the image branch (via [14, 22, 38J) distance-aware manner. UNION [18] leverages appearance from 2D images cluster and distinguish between mobile and immobile objects.
Unlike these approaches, which depend on iterative self-training [43, 46] and do not consider physical properties of instance [18, 26, 40, 43, 46], our method is designed to produce physical-state-speciff annotations and to alleviate the need for iterative refinement.
3 Method
This section explains the role and design choices of each module in our proposed automatic 3D bounding box annotator, OpenBox. As shown in Fig. 2, our system consists of two main stages: (1) Cross-modis Instance Alignment and (2) Adaptive 3D Bounding Box Generation.
3

---

Point Cloud Claster
HDBBSCan
Ground
LIDAR Point Clouds
Context-awaae
Cycllst,
Text Prompis
Instance- Point Cloud
Cloud
Groundin DINO
SAM
Multi-vien & Muld-drame Images
(Mask, ID)
Pretmined Model
Figure 3: Cross-modal Instance Alignment. To obtain refined point clouds, the pipeline generates two complementary proposals. The LiDAR (upper) branch removes ground points and applies HDBSCAN [3] to produce coarse 3D clusters. The image-LiDAR (lower) branch uses Grounding DINO [22] followed by SAM2 [31] to generate 2D instance masks. This information is unprojected into point clouds. Context-aware refinement fuses the two proposals. discarding noisy points and incorporating adjacent points from these clusters, yielding refined per-object point clouds.
3.1 Cross-modal Instance Alignment
Instance-level Feature Extraction. As shown in Fig. to leverage the strong capabilities of vision foundation models [22, 31] trained on large-scale datasets, we define a 3D-to-2D mapping function II, that projects the 3D point cloud p(t) E Mxx captured at time onto the j-th camera image
I 3 Using 2D detector [22] vp and segmentation model [31] o we obtain 2D boxes B class labels c)), masks M and instance IDs T, from 1 as follows:
B (I("),, M. = (BB)) tzt), DIt -I, 19)
(1)
where H and W denote the image height and width, B the 2D bounding boxes, D the pixel coordinates of p(t) projected onto 1/", and txt the text prompts. By associating each projected 3D point with its corresponding image pixel and mask label, obtain the instance-level point clouds - E
ij is i-th instance-level point cloud, which contains 3D coordinates (z, y, 2), semantic class, instance presence, and instance ID. However, the boundaries of the masks obtained from [31] are imprecise, due calibration errors, directly unprojecting them into 3D can result in noisy point clouds. To mitigate this issue, we adopt the adaptive erosion proposed in [13], which erodes masks based object size eliminate boundary noise while preserving instance structure. For convenience, we omit the subscripts and j from this point onward.
Here, F((()
Context-aware Refinement. As shown in Fig. 4-(c), LiDAR points are often projected onto background regions (e.g. guardrail and wall) that occlude the actual foreground instance, resulting in inaccurate unprojection. These noisy points, located outside the true object region, tend to yield 3D bounding boxes that are improperly scaled. To address this issue, we refine the unprojected instance- level point clouds F. We perform majority voting within clustered regions {R1, R2, R }. where each cluster R₂ IR x3 is obtained from the ground-removed raw LiDAR point clouds P using HDBSCAN [3], following ground removal based on [17]. For each segment RR, we compare with all instance-level point clouds F; and compute bidirectional proximity-
inclusion ratios. Specifically, we determine the proportion of points in R₂ that overlap with any point in F, and vice versa. If mutual overlap between the two clusters sufficient, the cluster R₂ is assigned the instance ID i and retained; otherwise, it is discarded. This process can be formulated as follows:

---

(a) Reference Image (
Cloud Instance-leeel Point Cloud (FF) Refined Point Cloud (Frr))
Figure 4: Context-aware Refinement. (a) Reference image. (b) Point cloud clusters R after using HDBSCAN [3] on ground-removed LiDAR point clouds. (c) Noisy instance-level point clouds F. (d) Result of the Context-aware refinement Fref-
{p Rx dist(p, < <8]]_ > a,
Kp E R2 dist(p, F.) < 63
> =
RS t (2)
R&
[F
where |. denotes the cardinality of set, and F.) < 5 holds if and only if there exists f E F,
such that 1P - f < 5.
3.2 Adaptive 3D Bounding Box Generation
Most prior methods [13, 43, 46] generate boxes without considering the physical properties of individual objects. This often leads to inaccurate localization and reduces the consistency of the data used to train 3D object detection networks. To address this issue, we propose adaptive box generation strategy that accounts for the physical types of each instance.
Static & Dynamic Points Decomposition. The refined instance-level LiDAR point clouds Fes obtained the previous step remain sparse. Yet, by aggregating consecutive point cloud frames to a global coordinate system, the instance-level point clouds can be significantly densified. Incorporating point clouds from dynamic objects is challenging, as this may introduce motion artifacts [40]. Thus, use the PP score [43] to estimate the ephemerality of each point in the refined point cloud.
Initial Bounding Box Generation. Empirically, we found that categorizing each instance based on its physical properties leads better performance. In particular, divide instances Fist into three types: rigid and static F rigid and dynamic FD. and deformable f For each type, we generate corresponding 3D bounding box. We use ChatGPT [27] to determine the object type based on the given semantic class to distinguish between rigid and deformable objects. Then, using this classification in conjunction with motion cues estimated via the PP score [43], we generate an appropriate 3D bounding box for each instance. We initially generate bounding box for all three object types using an approach [47] that maximizes the closeness of points to edges. However, due to the sparsity of the point clouds and occlusion, the resulting bounding box may underestimate the actual object size. To address this, we use ChatGPT [27] to retrieve the typical size of the object class terms of length, width, and height. If any of the initial bounding box dimensions are smaller than 80% of the typical size, we adjust the box size as described in the following sections.
Handling Static Rigid Instances. Although we densify the aggregated static instance point cloud F it still contains noise due to limitations of the context-aware refinement. To suppress it, we apply a surface-aware filtering method based on proximity voting over mesh vertices. Specifically, we reconstruct a mesh surface S from the point clouds using SDF [36]. For each vertex v E S, we identify nearby foreground and background points using the proximity function (c v) defined in Eq. where denotes the distance between the point and the mesh vertex. We retain vertices where foreground associations dominate, forming the refined surface S as defined in Eq. 4. The final refined static point cloud ref filtered surface vertices.
S2)) is then constructed by collecting all foreground points near the
Po(F,u) {pe F lpp u112 < <],
(3)
S {uES [Po 1) >>>>> (Po w)]}, 55 U
PC
(F u).
(4)
veSu
5

---

Surface-awane refinement
(b) 3D-2D ToU alignment
Maakkgondd Sfr
Vertex-level voting
Curmann Stmanccc thom
X kioee
Projected
Gnazndiu Buznding
TD Sunkee 5
Plond 2
Flieraniyy poinrt
Suffeee Sud
Bounding Contitsion
Handling Static & Rigid Instances. (a) We filter noisy points in the aggregated static point clouds via vertex-level voting on the reconstructed surface, producing Fi5(3) and Siet- (b) We then adjust the bounding box using surface normals and statistical priors, and select the final box based on 2D IoU between projected boxes and Grounding DINO [22] boxes.
Figure 5:
To refine the initial bounding box, extract the corresponding instance-level surface mesh Sins from S. As shown in Fig. 5, if the box is too small, we determine the resizing direction using surface normal vectors, rather than searching over directions in [13]. We rotate the surface into the ego coordinate system. Then, we compute the dot product between surface normals and the four orthonormal reference vectors determine which sides of the object are represented. If all four sides are covered, no resizing is needed. Otherwise, we generate two resized box candidates based statistical priors. This is because the longer side of the initial box cannot be reliably assumed to represent the object's 's actual length. (Please see Sec. C.3 for more details.) To select the optimal box, match each 3D candidate with 2D bounding boxes (from Sec. 3.1) using instance ID, project them onto multiple views and time steps, and compute their 2D IoUs. The box with the higher IoU is selected as the final result.
Handling Dynamic & Rigid Instances. In these cases, we rely on point clouds from a single moment in time, which makes it harder to accurately estimate the object's 's position, orientation, and size. deal with this problem, OpenBox uses the fact that the ori- entation of a dynamic object is approximately aligned with the direction of the position difference in adja- cent frames. OpenBox first estimates the object's orientation by the direction of the object trajectory associated with 2D tracking IDs. We then rotate the initial bounding box to align it with the estimated orientation angle. After aligning the orientation, we refine the box size. For each face along the X and Y axes, we compute the dot product between the out- ward surface normal and the LiDAR ray direction the face center. As shown in Fig. 6, OpenBox extends box only when the dot product between the ray and face normal is negative. We determine the final box size using standard object-size statistics.
Case
Case
Ini box
LIDAR
Box noemral
Bos extenine
Figure 6: Visibility -based box extension. Case 1 has one negative value from the dot product between the ray and the normal, yield- ing one-sided extension, whereas Case has two negative values, leading to two-sided extension.
Handling Deformable Instances. Deformable instances such as pedestrians, animals, cyclists exhibit articulated or non-rigid motion, causing spatial inconsistencies across frames that often lead to ghosting or distorted geometry when aggregated [4]. Due to their limited surface structure, geometry-ban -based refinement is ineffective. To maintain reliability, we generate bounding boxes from single frame by tightly fitting the visible region using the closeness-to-ede algorithm [47], which provides robust representations without relying rigid geometric assumptions.

---

Table 1: 3D object-detection results on the WOD [33] validation set. * 'ndicates trained and evaluated in the camera- camera-frustum region, while others use full 360 coverage. i and & denote models trained with CST and CBR from CPD [40], using the training settings given the next sentence. For F, we flip the OpenBox annotations and point clouds to obtain 360" coverage; for E, we fill the region outside the camera frustum with CPD annotations. All values denote AD3D at each IoU threshold. Bold means best performance, underlined means second-best. Only LI results are shown here; we provide the full L2 results in the Table
Vchicle
Pedestrian loUo, loUo.s
Cyclist loUb.3 loUo..
Method
Modality
loUo.5 / loUo.7
CPD* [40]
LiDAR
30.30 20,90
14.28 11.22
3,47
3.08
OpenBox* (Ours)
LiDAR Camera
70.49 / 32.41
57.95 17.11
20.81
2.15
DBSCAN [7]
LiDAR
2 0.29
0.51 0.00
0.28/
0.03
MODEST [43]
LiDAR
18.51 / 6,46
11.83/ 0.17
1.47 / 1.14
OYSTER [45]
LiDAR
30.48/ / 14.66
/ 0.18
1.27
0.33
CPD [40]
LiDAR
57.79 / 37,40
21.91 / 16.31
5,83 / 5,06
OpenBox (Ours)
LiDAR + Camera
66.89/ / 39.14
55.71 / 37.82
21.00 / 7.08
OpenBox3 (Ours)
LiDAR Camera
59.09 / 40.68
39.09 18.16
8.27 / 6.23
Human Annotation
93.31 / 75.70
87.25/77.95
58.84 54.88
Table 2: 3D object-detection results on the Lyft [12] validation set. Following [46], we evaluate in class-agnostic manner IoU = 0.25, and each value represents AP / APan. Bold means best performance, underlined means second-best.
Method
Modality
0-30m
30-50m
50-80m
0-80m
MODEST-PP (T'=00 0) [43] LiSe (I' 0) [46]
LiDAR
46.4/45.4
16.5 10.8
0.9/ 0,4
21.8/ 18.0
LiDAR + Camera
54.5/
24.2 22.8
1.4 1.2
29.2 27.5
OpenBox (Ours)
LiDAR + Camera
62.4/ 62.3
56.6 / 50.6
19.9 / 19.5
49.6 43.3
Human Annotation
82.8 / 82.6
70.8 / 70.3
50.2 / 49.6
69.1
4 Experiments
4.1 Experimental Setup
Dataset and Implementation Details. We conduct experiments on Waymo Open Dataset
(WOD) [33], Lyft Level 5 Perception Dataset (Lyft) [12]. and nuScenes [2]. For 3D object de- tection networks,
train Voxel R-CNN [6] for WOD [33], PointRCNN [32] for Lyft [12] and CenterPoint [42] for nuScenes [2] following the baselines [18, 40, 43, 46). We refer readers to WOD [33], Lyft [12] and nuScenes [2] for details of the evaluation metries. Our code is based on OpenPCDet [35] and MMDetection3D [5]. Additional details on training, hyperparameters, and
network architecture are provided in Appendix B.
Baselines. In the WOD [33] benchmark, the state-of-the-ard method CPD [40] evaluates the relia- bility of 3D bounding boxes using the CSS score and constrains network training by jointly learning dense CProtos within those boxes. For the Lyft [12] dataset, LiSe [46] fuses 3D bounding boxes from an image branch [38] and a LiDAR branch [43] based on distance. Finally, for nuScenes [2], UNION distinguishes mobile objects by leveraging visual appearance features extracted with DINOv2 [28]. Since our method in WOD [33] performs annotation only on point clouds that fall within the camera frustum field of view (FOV), we conduct experiments under two different settings.
Experimental Scenarios. We conduct experiments under two scenarios to comprehensively evalu- ate the quality of our automatic annotations. Scenario / trains a 3D object detector automatically annotated data and evaluates it on a human-annotated validation dataset. Scenario directly compares the automatic annotations with the human annotations the training set.
7

---

Table 3: Annotation performance on Lyft [12] training dataset. We evaluate our automatically annotated dataset with -anan-andtaated dataset. Following [46], we evaluate in class alass-agnostic manner at loU 0.25, and each value represents 3P3D. Bold means best performance.
Method
0-30m
30-50m
50-80m
0-80m
LiSe [46]
17.47
6.87
1.35
6.31
OpenBox (Ours)
56.62
28.10
6.47
26.25
Table 4: 3D object-detection results on the nuScenes [2] validation set. Following [18], we evaluate for 3 classes, and each value represents AP3D. Bold means best performance.
Method
Modality
Car
Pedestrian
Cyclist
UNION [18]
LiDAR + Camera
30.1
41.6
0.0
OpenBox (Ours)
LiDAR + Camera
40.9
62.7
5.2
4.2 Main results
Comparison on WOD. Table presents the LEVEL_ 1 AP3D results of experiments conducted on the WOD [33] under Scenario 1. For fair comparison with the tate-of-theeart method CPD [40], we conduct the experiments under two FOV (Field of View) settings. Under the camera-frustume method consistently outperforms CPD [40] for vehicle and pedestrian classes, though CPD incorporates additional training techniques (e.g.. CST and CBR) beyond its annotation pipeline. The inferior performance of the cyclist class be attributed our of the prompt "bicycle" in Grounding DINO [22]. which often yields undersized bounding boxes compared to those enclosing the entire cyclist. Furthermore, we evaluate two extended settings: (1) applying CPD's [40] training schemes (CST and CBR) to our boxes, and (2) combining our boxes with CPD's [40] and then training with CST and CBR. Both approaches lead performance improvements across all classes, with particularly notable gains for pedestrian and cyclist categories. We attribute this to fundamental design difference: CPD [40] annotates only stationary objects, resulting in low recall. Furthermore, it relies on class-agnostic tracking and classifies based on box-size statistics. In contrast, our method identifies the object class using a 2D vision foundation model [22], and generates adaptive bounding boxes that reflect each class's 's physical properties, resulting in more accurate annotations.
FOV setting,
Comparison on Lyft. Table shows the results of class-agnostic 3D object detection on the Lyft [12] dataset under Scenario 1, using an IoU threshold of 0.25. To ensure fair comparison, we evaluate against baseline methods [43, 46] that do not assume multiple traversals and do not apply self-training strategies. Our method demonstrates improved performance both APBEY and AP3D across all distance ranges compared to baselines. In particular, for long-range scenarios (50-80m), our method outperforms LiSe [46] by +18.5% APBEV and + +1.44% in AP3D. Furthermore, as shown in Table 3, we evaluate the performance of automatic annotations in the Scenario 2 environment. Our method consistently outperforms LiSe [46] across all ranges. This performance gap arises because LiSe [46] integrates 3D boxes from the image branch, generated using the method of [38], and from the LiDAR branch, based on [43]. However, neither of these components explicitly considers the physical properties or semantics of the object classes, which limits their precision.
Comparison on nuScenes. As shown in Table 4, we observe performance improvements across all classes under Scenario 1. One key reason for the significant gains is that, unlike OpenBox, UNION [18] omits the refinement process for point clouds and 3D bounding boxes. In particular, it does not explicitly consider the camera-lidar calibration error when projecting LiDAR point clouds on DINOv2 [28] feature maps which leads to noise at the boundary of the objects. Additionally, UNION [18] neither resizes nor relocalizes the initial 3D bounding boxes, leading the model to predict suboptimal bounding boxes.
8

---

(a)
(A)
(b) CPD
(A)
(4)
MITHLO O
(A)
Figure 7: Comparison of automatic annotation on WOD [33] training set. Each row compares automatically annotated boxes with human-annotated boxes, while each column corresponds to different scene. Blue boxes represent the automatically generated boxes, and red boxes indicate the human-annotated boxes. We visualize CPD [40] annotations filtered by a minimum CSS score threshold. Best viewed in color and zoomed in.
Table 5: Ablation study on the Vehicle class for WOD [33] training set. Surface-aware, Context- aware, and 3D-2D IoU refer Surface-aware Refinement, Context-aware Refinement, and 3D-2D IoU alignment respectively. All results stand for the Vehicle class using AP3d at IoU 0.4.
(a) Point-level refinement
(b) Box-level refinement
Surface-award aware Context-aware AP3D
Visibility-baad based 3D-2D loU
AP3D
30.34
30.49
32.52
34.71
38.65
38.65
4.3
Ablation study
To analyze the impact of each module on automatic annotation, we conduct an ablation study under the Scenario In Table 5-(a), observe that applying both point-level refinement modules yields the highest performance. This is because Context-aware Refinement is applied to all instances regardless of their physical properties, whereas Surface-aware Refinement is specifically designed for rigid and static instances. Furthermore. some effects of Surface-aware Refinement are partially covered by Context-aware Refinement, which explains its relatively larger contribution when applied alone. Similarly, Table 5-(b) presents an ablation study focusing on modules that contribute to box-level refinement. The 3D-2D IoU alignment module designed to resize and relocate boxes for static and rigid instances, while the visibility 3D-2D loU alignment has greater overall impact when combined with the visibility-basse method for two main reasons: (1) the number of static vehicles in the WOD [33] dataset is significantly larger than that of dynamic vehicles, and (2) the two candidate boxes considered by 3D-2D IoU alignment typically differ by 90%, leading to more substantial effect on the AP3D
module is applied to dynamic and rigid instances.

---

Text Prompt: Stroller
Text Prompt: Fire Hydrant
Text Prompt: Dog
Figure 8: Our automatic annotation results on novel classes in WOD [33]. In this visualization, red points represent instance-level point clouds, while cyan boxes indicate the automatically generated annotations. Best viewed in color and zoomed in.
4.4 Qualitative Result
As shown in Fig. 7, we compare 3D bounding boxes from automatic and human annotations. Overall, OpenBox shows higher precision and recall compared the baselines [40, 45J. Region (A) illustrates a static travel trailer. Since [40, 45] generate boxes without considering an instance's physical properties, it remains unannotated despite being static. In contrast, OpenBox recognizes it as static, enabling annotation even with sparse point evidence. In region (B), our approach successfully detects a static car and a pedestrian inside garage, which the baselines miss or mislocalize. This because our method refines the point cloud to isolate cettance-specfffe points. Region (C) shows that our method can automatically annotate vehicles on the opposite lane, even when no corresponding human annotations exist. Although both the baseline and our method detect these vehicles, ours localizes them more accurately by extending to both rigid and dynamic instances and by jointly leveraging vision foundation model, resulting in higher recall. In addition, OpenBox enables automatic annotation of open-vocabulan objects beyond the predefined classes in existing autonomous-drivimg datasets [2, 10, 12, 33, 39]. As shown in Fig. 8, it successfully annotates objects such strollers, fire hydrants, and dogs, which are essential to consider in real-world driving scenarios.
5
Conclusion
In this paper, we propose OpenBox, novel automatic 3D bounding box annotation pipeline. Open-- Box effectively leverages 2D vision foundation models to generate open-vocabulary 3D annotations. To ensure high-quality box generation, it refines instance-level point clouds and performs adaptive 3D bounding-box generation tailored to the physical properties of each instance. Our extensive exper- iments on diverse autonomous driving datasets demonstrates the superiority of the proposed method in annotation quality over prior baselines, while our comprehensive ablation study substantiates the effectiveness of each component in the annotation pipeline. We hope that our method can contribute to future research on foundation models for 3D perception.
Limitations. Adverse weather reduces contrast and obscures edges, which makes 2D vision models unreliable. The 3D annotations built on those models inherit the errors and often miss instances, resulting in imprecise box boundaries. Deformable categories such as pedestrians and cyclists also suffer because pose variation makes full-extent inference unstable. The method then falls back to fixed class-level sizes, which frequently under- over-size the box. At long range, LiDAR returns become too sparse to constrain geometry, so box fitting is ill-conditioned and localization becomes less precise.
6 Acknowledgments
This work was supported by IITP grant (RS-2021-1121133: AI Graduate School Program (Seoul Na- tional University) (5%), No.2002-0-0200 AI Innovation Hub (10%), 25-InnoCORE-O1: InnoCORE program of the Ministry of Science and ICT (10%)) and NRF grant 2023RIAAIC20078 (75%)) funded by the Korea government (MSIT).
10

---

References
[1] Stefan Baur, Frank Monsmann, and Andreas Geiger. Liso: Lidar-only self-supervised 3d object detection.
In Proc. European Conference on Computer Vision (ECCV), 2024.
[2] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancario Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proc. JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[3] Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. Density-based clustering based on hierarchical density estimates. In Jian Pei. Vincent S. Tseng, Longbing Cao. Hiroshi Motoda, and Guandong Xu, editors, Advances in Knowledge Discovery and Data Mining, 2013.
[4] Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, B. Ivanovic, Or Litany, Zan Gojeic, S Fidler, Marco Pavone, Li-Na Song, and Yue Wang. Omnire: Omni urban scene reconstruction. In International Conference Learning Representations (ICLR), 2025.
[5] MMDetection3D Contributors. MMDetection OpenM aa next-generation platform for general 3D object detection, pastps://gitin com/ htte//ppenmmmmmaaee mmlab/ httpb//meeeeeeiinndddd 2020.
[6] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cmn: Towards high performance voxel-basee 3d object detection. In Proceedings of the AAAI conference on artificial intelligence (AAAI), 2021.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. density-hased algorithm for discovering clusters large spatial databases with noise. Proc. Second International Conference on Knowledge Discovery and Data Mining (KDD), 1996.
[8] Djamahl Etchegaray, Zi Huang, Tatsuya Harada, and Yadan Luo. Find n' propagate: Open vocabulary 3d object detection in urban scenes. In Proc. European Conference on Computer Vision (ECCV), 2024.
[9] Martin A. Fischler and Robert C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun, ACM, 24(6), 1981.
[10] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proc. IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR). 2012.
[11] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling ooca-vocyy image segmentation with image-level labels. In Proc. European Conference on Computer Vision (ECCV), 2022.
[12] John Houston, Guido Zuidhof. Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska. One thousand and one hours: Self-driving motion prediction dataset. In Conference on Robot Learning. PMLR, 2021.
[13] Rui Huang, Henry Zheng, Yan Wang, Zhuofan Xia, Marco Pavone, and Gao Huang. Training an open- vocabulary monocular 3d detection model without 3d data. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[14] Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proc. IEEF/CVF International Conference Computer Vision (ICCV), 2023.
[15] Linh Kästner, Vlad Catalin Frasineanu, and Jens Lambrecht. A 3dd-deep-leartnng-bass based augmented reality calibration method for robotic environments using depth sensor data. IEEE International Conference on Robotics and Automation (ICRA), 2020.
[16] In-Jae Lee, Sihwan Hwang, Youngseok Kim, Wonjune Kim, Sanmin Kim, and Dongsuk Kum. Crah: Camera-radar fusion for reducing depth ambiguity in backward projection based view transformation. In IEEE International Conference Robotics and Automation (ICRA), 2025.
[17] Seungjac Lec, Hyungtae Lim, and Hyun Myung. Patchwork++: Fast and robust ground segmentation solving partial under-segmentation using 3d point cloud. Proc. IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), 2022.
[18] Ted Lentsch, Holger Caesar, and Dariu M Gavrila, UNION: Unsupervised 3D object detection using object appearance-boe based pseudo-classes. classes. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
11

---

[19] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang. Lei Zhang, and Jianfeng Gao. Segment and recognize anything any granularity. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Proc. Luropean Conference on Computer Vision (ECCV), 2025.
[20] Liunian Harold Li Pengchuan Zhang*, Haotian Zhang Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[21] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's- s-eye-vve representation from multi-camera images via spatiotemporal trans- formers. In Proc. Europea Conference on Computer Vision (ECCV), 2022.
[22] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. Proc. European Conference on Computer Vision (ECCV), 2024.
[23] Zhijian Liu, Haotian Tang, Alexander Amini, Xingyu Yang. Huizi Mao, Daniela Rus, and Song Han. Bev- fusion: Multi-task multi-sensor fusion with unified s-ey view representation. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[24] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann. Dirk Weissenbom, Alexey Dosovit- skiy, Aravindh Mahendran, Anurag Arnah, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection. In Proc. European Conference Computer Vision (ECCV), 2022.
[25] Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R. Qi, Xinchen Yan, Scott Ettinger, and Dragomir Anguelov. Motion inspired unsupervised perception and prediction in autonomous driving. In Proc. European Conference on Computer Vision (ECCV), 2022.
[26] Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R. Qi. Xinchen Yan. Scott Etinger, and Dragomir Anguelov. Unsupervised 3d perception with 2d vision-language distillation for autonomous driving. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[27] OpenAL. ChaiGPT: language model (Mar 2024 version). https://chatt opw.ai..ecacooooom. 2024. Accessed: 2024-05-09.
[28] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
[29] Mihir Prabhudesai, Hsiao-Yo Yu Fish Tung. Syed Ashar Javed, Maximilian Sieb, Adam W Harley, and Katerina Fragkiadaki. Embodied language grounding with 3d visual feature representations. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transfcrable visual models from natural language supervision. In Proc. International Conference on Machine Learning (ICML), 2021.
[31] Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos, 2024.
[32] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proc. IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[33] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In Proc. IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). 2020.
12

---

[34] Zineng Tang. Long Lian, Seun Eisape, XuDong Wang, Roei Herzig, Adam Yala, Alane Suhr, Trevor Darrell, and David M. Chan. Tulip: Towards unified language-image pretraining, 2025.
[35] OpenPCDet Development Team, Openpedet: An open-source toolhox for 3d object detection from point mm mww.ab/ppee/Peee...... 2020.
clouds. https://githiei
[36] Ignacio Vizzo, Tiziano Guadagnino, Jens Behley, and Cyrill Stachniss. Vdbfusion: Flexible and efficient Isdf integration of range sensor data. Sensors, 2022.
[37] Daniel Weher, Enkelejda Kasneci, and Andreas 7ell. Exploiting augmented reality for extrinsic robot calibration and eye-based human-robot collaboration. In Proc. ACM/IEEE International Conference on Human- Robout Interuction, HRI '22, 2022.
[38] Yi Wei, Shang Su, Jiwen Lu, and Jie Zhou. FGR: Frustum-Aware Geometric Reasoning for Weakly Supervised 3D Vehicle Detection. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[39] Benjamin Wilson, William Qi. Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self- oeff-drivin perception and forecasting. In Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), 2021.
[40] Hai Wu, Shijia Zhao, Xun Huang, Chenglu Wen, Xin Li, and Cheng Wang. Commonsense prototype for outdoor unsupervised 3d object detection. Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[41] Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang, Eda: Explicit text-decoupling and dense alignment for visual grounding. Proc. JEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[42] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proc. IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[43] Yurong You, Katic Luo, Cheng Perng Phoo, Wei-Lun Chao, Wen Sun, Bharath Hariharan, Mark Camphell, and Kilian Q. Weinberger. Learning to detect mobile objects from lidar scans without labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[44] Hu Zhang, Jianhua Xu, Tao Tang, Haiyang Sun, Xin Yu, Zi Huang, and Kaicheng Yu. Opensight: A simple open-wocabulary framework for lidar-based object detection, In Proc. European Conference Computer Vision (ECCV), 2024.
[45] Lunjun Zhang, Anqi Joyee Yang. Yuwen Xiong, Sergio Casas, Bin Yang, Mengye Ren, and Raquel Urtasun. Towards unsupervised object detection from lidar point clouds. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[46] Ruiyang Zhang, Hu Zhang, Hang Yu, and Zhedong Zheng. Approaching outside: Scaling unsupervised 3d object detection from 2d scene. Proc. European Conference on Computer Vision (ECCV), 2024.
[47] Xiao Zhang, Wenda Xu, Chiyu Dong, and John M. Dolan. Efficient I-shape fitting for vehicle detection using laser scanners. In IEFE Intelligent Vehicles Symposium (IV), 2017.
[48] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühh, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. Proc. European Conference on Computer Vision (ECCV), 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce contributions mrooss Instance Alignment and Adaptive Bounding Box Generation) in the abstract and introduction part.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our method in the Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our method does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We contain configuration of implementation details the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: Code will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips wwtppbccc//aaeeccccccccccccc guwdes/de/eee wwweeeeeeeeeebbaeeeeeeee ssssonolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide the information of dataset used for the experiment. We also include the additional details in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We could not because of the limited computing resource.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include those information the appendix due to page limit.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics atttps:/neeus http:p:////errp...ppppplaalaaa/ htc//publc//Etiiie csGuidel
de ines?
Answer: [Yes]
Justification:
We read and understand the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Our research is related social impact in autonomous driving area.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe that no risk is involved.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited the reference paper, code and data properly.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
We will release the well-documented assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We did not conduct any research related to crowd sourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not conduct any research related to crowd sourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use ChatGPT [27] for part of our method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: for what should or should not be described.
20

---

Appendix
A Overview
In this supplementary material, we provide additional details of OpenBox. Appendix B describes the experiment details. Appendix provides a detailed approach for height refinement and surface estimation. We further present more experimental results in Appendix D.
B
Implementation Details
Table 6: Training and network details for experiment
configs
Voxel R-CNN [6]
Point RCNN [32]
CenterPoint [42]
optmizer
Adam W
Adam W
AdamW W
base learning rate
le-2
1e-2
le-4
weight decay momentum
le-3
1e-2
1e-2
0.9
le-2
momentum range
[0.95, 0.85]
[0.95, 0.85]
learning rate decay learning rate clip gradient norm clip batch size epoch
0.1
le-7
le-7
10
10
35
16
2
32
20
60
20
We train models [6. 32, 42] 8 NVIDIA A6000 GPUs (48GB) and 2 AMD EPYC 7763 CPUs. We also employ VDBFusion [36] for SDE the Context-aware Refinement part (see Sec. 3.1), the hyperparameters a, B, and are to 0.3, 0.2, and 0.1, respectively. In addition, the threshold 7 used the Handling Static & Rigid Instance part (see Sec. 3.2) is set to 0.15.
C
Additional details for method
C.1
ChatGPT prompt
To obtain 3D bounding box size priors and determine the rigidity of objects, we utilized the following prompts with ChatGPT-4 [27]:
Prompt: Please provide the typical 3D bounding box size for [class]
Response:
Here are the typical 3D bounding box dimensions (Length Width Height in meters) for [class], based common Waymo Open Dataset.
datasets like nuScenes, KITTI,
Prompt: Is [class] deformable rigid?
Response:
[class] is considered a deformable rigid object.
Below the list of categories we provided GroundingDINO [22] as text prompts:
[Car, Bus, Person, Truck, Construction Vehicle, Trailer, Barrier, Bicycle, Motorcycle, Traffic Cone, Dog, Fire Hydrant, Stroller]
C.2 Height Refinement
In section 3.1, we exclude points whose z-coordinates are below a predefined threshold remove remaining ground points after the RANSAC [9] based ground removal. This preliminary step can lead
21

---

Bird's Eye View
Surfice nomal
Case I
Onhonurmal Vectur
Case 2
Case 3
(b)
Figure 9: Illustration of Surface Estimation.
to bounding boxes appearing elevated above actual object. To address this issue, we propose an additional refinement step accurately estimate the z-coordinate of each bounding box. Specifically, given an instance with length and width w, we calculate a radius as:
L
(5)
2
We then define the set of points within this radius from the ego-position Pego of the instance:
p' = {p E |Pego p/2 < Lb),
(6)
where P denotes the point cloud corresponding to the frame in which the instance is located. After sorting points in P' by their z-coordinate in ascending order, we select the z-coordinate at the 1%, effectively removing potential noise and LiDAR reflectance outliers near the ground. This procedure ensures robust estimation of the ground level near the instance.
C.3 Surface Estimation
We determine the surface direction of instance-level surfaces Sins to facilitate 3D-2D loU alignment, as described in the Handling Static & Rigid Instance section (Sec. 3.2). Specifically, we compute the dot product between the normal vectors of Siss and set of four orthonormal reference vectors to identify the surface direction.
As illustrated in Fig. 9, in (a), only the direction yields a dot product greater than the predefined threshold y 0.8, allowing in (c), 1 2, and 4 all surpass the threshold, indicating the presence of multiple surface orientations.
identify the surface direction. In (b), both @ and @ exceed y, while
D More Experimental Results
Quantitative Results Table presents the AP3p results under the LEVEL_2 2 of the WOD [33] validation split. Compared to the results under the LEVEL_1 criterion shown in Table 1, which reflects performance in easier scenarios, the overall performance lower. Nevertheless, our approach outperforms other baselines [7, 40, 43, 45], indicating that the proposed dataset annotations enable the 3D object detection network [6] learn effectively even under more challenging conditions.
We demonstrate OpenBox on the WOD [33] dataset in two scenarios. Scenario 1 compares annotation quality on the original WOD training for vehicle, pedestrian, and cyclist classes. Scenario 2 presents annotation results for novel object categories. A detailed visualization of both scenarios is provided in the attached video (supple_ wiieeeeviee_.eee .mp).
Qualitative Results
22

---

Table 3D object-detection results on the WOD [33] validation set. Models marked with * are trained and evaluated in the camera- frrssamm region, while others use full 360 coverage. and denote models trained with CST and CBR from CPD [40], using the training settings given in the next sentence. For i, we flip the OpenBox annotations and point clouds to obtain 360 coverage; for we fill the region outside the camera frustum with CPD annotations. All values denote 3D at each IoU threshold for LEVEL_2. 2. Bold means best performance, underlined means second-best.
Vchicle
Pedestrian IoUo.3 / IoU0.5
Cyclist IoUn.3 IoUn.5
Method
Modality
IoUo.s / IoUo.7
CPD* [40]
LiDAR
26.09/ 17.91
11.87 9.30
3.34/
OpenBox* (Ours)
LiDAR Camera
62.74 / 28.03
51.55 15.06
20.08 1.88
DBSCAN [7]
LiDAR
1.94 0.25
0.19/ 0.00
0.20/
0.00
MODEST [43]
LiDAR
15.83 / 5.48
8.96/ 0.10
0.43/
0.20
OYSTER [45]
LiDAR
26.21 14.10
3.52/ 0.14
1.24/ 0.32
CPD [40]
LiDAR
50.18 / 32.13
18.01 13.22
5.61
4.87
OpenBoxt (Ours)
LiDAR Camera
58.42 / 33.72
47.78 31.77
20.19
6,81
OpenBox: (Ours)
LiDAR Camera
51.70/ / 34.95
33.02/ 23.50
7.95
5.99
23


Reviews:
[{"review_id": "DuVqb2dyeR", "reviewer": "Reviewer_WBAe", "summary": "This work proposes an auto-labeling pipeline for annotating 3D bounding boxes by leveraging 2D visual foundation models. The pipeline first segments object instances using visual knowledge distilled from foundation models and aggregated multi-frame LiDAR point clouds. It then generates geometrically adaptive bounding boxes by incorporating category-specific prior knowledge, such as dynamic/non-rigid properties and real-world size distributions. The proposed approach produces high-fidelity annotations without requiring iterative self-training, demonstrating significant potential for scalable 3D dataset generation.", "questions": "1. Are there alternative methods for refining the bounding box fit to the segmented instance point clouds? Additionally, what is the quantitative impact on detection precision (e.g., mAP, IoU) if bounding box fitting is omitted from the pipeline?\n2. Which object categories present the most significant challenges for accurate estimation within this pipeline? Furthermore, could representative failure cases or limitations be discussed to provide deeper insight.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.This auto-labeling framework has potential value on open-world perception as annotating 3D instance is very expensive.\n2.The method accounts for diverse object properties, including rigid, non-rigid, and dynamic objects, making it applicable to a wide range of real-world scenarios.\n3. The pipeline does not relying on self-training which makes it more practical for deployment.\n\nWeakness:\n1. Since the point clouds only encode the partial view of an object, how the bounding-boxes are adjusted to fit the instance. It seems that the most critical part is using PP Score and point-to-edge closeness for bounding-boxes adherence. It is better to illustrate more in case the reader do not familiar with the cited work.\n2. The bounding box extension based on the normal and ray direction is not general. The deformation of real-world objects can be in any form.\n3. The orientation estimation on the static objects remains unclear.", "was_revised": true, "final_justification": "The rebuttal effectively addresses my concerns. Although the techniques used in this work are not novel, it demonstrates significant practical value for industrial applications. I am inclined to accept this work and look forward to its open-sourcing.", "rebuttal": "We sincerely appreciate the reviewer's positive feedback on the novelty and practicality of our proposed framework. We are encouraged to see that the reviewer recognizes: (1) the value of automatic labeling for 3D perception, especially given the high cost of 3D instance annotation in open-world settings (2) the generality of our method, which explicitly handles diverse object types including rigid, non-rigid, and dynamic objects enabling wide applicability across real-world scenarios and (3) the fact that our pipeline does not rely on self-training, which improves its practicality and robustness for real-world deployment.\n\n**Q1. Are there alternative methods for refining the bounding box fit to the segmented instance point clouds? Additionally, what is the quantitative impact on detection precision (e.g., mAP, IoU) if bounding box fitting is omitted from the pipeline?**\n\nWe agree with the reviewer’s concern that the bounding box extension based on surface normals and ray directions is not universally applicable. As noted in our main paper (lines 283–285), we acknowledge the limitations of our approach, especially when handling deformable classes. The primary reason we adopted this extension method is that we assume rigid objects, making this strategy suitable for the majority of cases. To address the reviewer’s concern and better handle segmented instance point clouds, we believe a more robust refinement could be achieved by first increasing point cloud density through instance-level registration, utilizing instance IDs obtained from SAM2 [3]. Then, using point-aware bounding box optimization with techniques such as Ray Tracing Loss [6] and Point Ratio Loss [6], along with the Closeness-to-Edge criterion [9], it would be possible to more accurately refine bounding boxes for deformed static objects. We consider this a promising direction for future work.\n\nAdditionally, in response to the reviewer’s suggestion, we conducted an ablation study to assess the impact of the bounding box refinement module, as presented in Table 1.\n\n**Table 1**. Ablation study of 3D bounding box refinement module on WOD [1] validation set. V, P and C indicate Vehicle, Pedestrian and Cyclist. AP 3D (Average Precision for 3D) for each classes are evaluated under IoU=0.7/0.5/0.5 respectively.\nVisibility-based is designed for refining dynamic rigid objects, while the 3D-2D IoU alignment targets static rigid objects for bounding box refinement.\n\n| | Visibility-based | 3D-2D IoU alignment | AP 3D ↑(V/P/C) |\n|:---:|:---:|:---:|:---:|\n| (a) | | | &nbsp; 2.95 /  &nbsp; 0.00 / 0.00 |\n| (b) | | ✔ | 29.97 / 13.25 / 0.94 |\n| (c) | ✔ | | 26.33 / 12.82 / 1.08 |\n| (d) | ✔ | ✔ | 32.41 / 17.11 / 2.15 |\n\nTable 1 illustrates the impact of the 3D bounding box refinement module on detection performance. We use Voxel R-CNN [2] as the 3D object detector, which is trained on an automatically annotated version of the Waymo Open Dataset (WOD) [1] using OpenBox, and evaluated on the human-annotated WOD [1], following setting 1 described in the main paper. Both training and evaluation are performed within the camera frustum region. When comparing (a) and (d), we observe improvements in AP 3D performance across all classes. This is because (a) generates sub-optimal 3D bounding boxes in regions where the LiDAR point cloud captures partial views due to either long-range distances or occlusion. And it causes the model to predict boxes which fail to reach the required IoU thresholds with the GT boxes used for AP 3D evaluation. Furthermore, when examining (b), (c), and (a) collectively, we see that (b) (Rigid Static) shows a larger performance gain from (a) compared to (c) (Rigid Dynamic). This is attributed to the fact that parked vehicles are more prevalent than moving vehicles in autonomous driving datasets. Moreover, static instances undergo aggregation, resulting in denser point clouds, and bounding boxes are generated after two stages of point-level filtering (context-aware and surface-aware). In contrast, dynamic instances do not benefit from aggregation and are filtered only by the context-aware module, leading to higher uncertainty.\n\n**Q2. Which object categories present the most significant challenges for accurate estimation within this pipeline? Furthermore, could representative failure cases or limitations be discussed to provide deeper insight.**\n\nWe regret that policy restrictions prevent us from including figures illustrating those failures and challenging cases. We will include the failure case in the supplementary material with explanation below.\n\nReviewing the quantitative results, annotating cyclists with 3D bounding boxes proves most challenging. This stems from imperfections of 2D visual foundation models such as GroundingDINO [8] and SAM2 [3]. Since GroundingDINO's [8] training did not include the prompt \"cyclist,\" it cannot infer that class. We therefore use the prompt \"bicycle.\" In the same scene, the model may first detect both bicycle and rider together and then detect only the bicycle (This inconsistency is clear in the supplementary video at 01:09, where a cyclist is mislocalized) As a result, the SAM2 mask size changes and the unprojected point cloud varies. Even with a size prior, these variations prevent accurate 3D bounding box annotation.\n\nWe identified three additional failure modes. First, distant objects sometimes are not detected by GroundingDINO [8], 3D bounding boxes cannot be generated **(01:53 \\\\(\\\\sim\\\\) 01:56 in the supplementary video, traffic-cone appears without consistency)**. Second, even after point cloud aggregation, distant objects remain too sparse and vanish during refinement. **(00:24 sec in the supplementary video, a distant static vehicle is omitted)**. Last, imperfections in the PP score [4] can misclassify dynamic objects as static, producing false positives. **(01:13 in the supplementary video, a false positive 3D box appears behind a purple vehicle annotation.)**\n\nTo mitigate afore-mentioned issues are as follows. First, using detection by tracking algorithms such as AB3DMOT [5] could provide a degree of consistency. Second, fusing image based 3D auto annotation methods such as OVM-Det3D [6] could address sparsity. Finally, leveraging scene flow (VoteFlow[7]) to predict object motion more robustly is expected to substantially reduce false positives.\n\n## Clarity ##\n\nWe will include the explanation of some related works before methodology part in the main paper for readers who are not much familiar with this area. Below is the explanation of PPscore [4] and point-to-edge closeness.\n\n**PP Score (Persistence Point Score) [4]:** Measurement of how \"persistent\" a LiDAR point is across multiple passes (traversals) of the same route. Higher means high probability of static point.\n\n**point-to-edge closeness [9]:** The closeness score measures how well a candidate rectangle hugs a point set by summing each point's inverse distance to its nearest rectangle edge. Rectangles with more points lying close to their sides get higher scores.\n\nTo determine the static rigid box orientation using the closeness criterion, we proceed as follows:\n\n1. **Sample candidate orientations:** We rotate the point set by a series of angles (e.g. 0°, 10°, 20°, … up to 180°).\n\n2. **Evaluate snugness at each angle:** For each rotated set, we fit the smallest axis-aligned rectangle around the points and compute the closeness score by summing each point's inverse distance to its nearest rectangle edge.\n\n3. **Select the optimal orientation:** The angle that yields the highest closeness score is chosen as the final orientation, since it indicates the rectangle that \"hugs\" the point cloud most tightly.\n\nThis procedure reliably identifies the best alignment by directly measuring how well the rectangle encloses the observed points.\n\n\n\n[1] Sun et al. (2020). Scalability in perception for autonomous driving: Waymo open dataset. In CVPR.  \n[2] Deng et al. (2021). Voxel R-CNN: Towards high performance voxel-based 3D object detection. In AAAI.  \n[3] Ravi et al. (2024). SAM 2: Segment anything in images and videos. arXiv.  \n[4] You et al. (2022). Learning to detect mobile objects from LiDAR scans without labels. In CVPR.  \n[5] Weng et al. (2020). AB3DMOT: A baseline for 3D multi-object tracking and new evaluation metrics. In IROS.  \n[6] Huang et al. (2024). Training an open-vocabulary monocular 3D object detection model without 3D data. In NeurIPS.  \n[7] Lin et al. (2025). VoteFlow: Enforcing local rigidity in self-supervised scene flow. arXiv.  \n[8] Liu et al. (2023). Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. arXiv.  \n[9] Zhang et al. (2017). Efficient L-shape fitting for vehicle detection using laser scanners. In IEEE IV (Intelligent Vehicles Symposium)."}, {"review_id": "xV1F8ftLZB", "reviewer": "Reviewer_2oE1", "summary": "In this paper, author come up with a noval two stage automatic annotation pipeline for autonomous driving called OpenBox. In the first stage, Cross-modal Instance Alignment is executed. OpenBox first utilizes the instance masks generated by 2D visual model and unproject them back to 3D point cloud to generate instance level point cloud. The results are then fused with the result of point cloud clustering using HDBSCAN. The second stage is Adaptive 3D Bounding Box Generation, which generates boxes on the basis of the point cloud from stage1 and the physical types of each instance. Compared with other automatic annotation methods, OpenBox requires no self-training and generates more accurate annotation using context-aware refinement and surface-aware noise filtering. Experiment results on Waymo show OpenBox-generated annotations train the best performaned model achieving 70.49% AP3D for the vehicle class and results on Lyft shows better performance compared with model trained on human annotation boxes.", "questions": "## Questions:\nAnnotation cost is pointed out in the abstract, and one of the contribution of OpenBox is that it requires no training of 3D object detectors. Typically 3D bounding box is decided by its center, size and rotation. The localization(center) of bounding box is substituted by 2D segmentation models and 3D point cloud clustering. Box size follows the typical size of the category and rotation is decided by 3D-2D IoU alignment. It is great to combine existing methods and provide amazing annotation effects. However shall we neglact the training cost of the existing methods like DINO, SAM2 and ChatGPT. Furthermore, what is the inference cost of all models. Are all the models/algorithms used in OpenBox use less resources/cost less time than a traditional 3D object detection? If possible, please provide these information in the appendix and we can have a better comparison between OpenBox and traditional 3D object detection way.\n\n## Suggestions:\n1. The author should provide the missing experiment settings and appendix.\n2. More details about the auto-annotation process are welcomed, like the inference cost(time/computation) of the methods/models used and the template to and the response from ChatGPT.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The experiment on WayMo and Lyft shows convincing evidence to the ability of OpenBox. With no more training required, OpenBox combines existing models (Grounding DINO, SAM2, ChatGPT) and point cloud processing methods(SDF, HDBSCAN) with noval postprocess on static, dynamic and deformable categories, providing a economical pipeline to perform high-quality anto-annotation task, helping improve the 3D object detection performance in autonomous driving. \n\nWeaknesses:\n- The paper fails to provide the detailed of Setting 1 and Setting 2 in section Experiments, which makes the whole section confusing.\n- The paper lacks details of some process. For example, in section 3.2 ChatGPT is used twice to decide the type of the object and retrieval the typical size of the object. It's better to provide some conversation examples. \n- The paper mentions about the appendix in Line 209 but no appendix is found.\n- The paper fails to provide a code to reproduce the experiment results in the paper", "was_revised": true, "final_justification": "The problems and weakness of the initial review is well explained in the rebuttal. And it is true we should collect and use the existing fundamental models rather than 'build new wheels'. Since the author proves the inference cost of OpenBox is same or less than traditional 3D object detection models, I have no major questions about the paper and improve the Clarity score from 1 to 2 and final score from 3 to 4.", "rebuttal": "We appreciate the reviewer's recognition of our results on Waymo and Lyft. As noted, OpenBox requires no additional training yet seamlessly integrates existing models (Grounding DINO, SAM2, ChatGPT) with point cloud techniques (SDF, HDBSCAN) and a novel postprocess for static, dynamic and deformable categories. This cost-efficient pipeline delivers high-quality auto-annotations and drives improvements in 3D object detection for autonomous driving.\n\nBelow, we provide detailed responses to the reviewer's questions.\n\n## Details about the automatic annotation process ##\n\n**ChatGPT prompt**\n\nTo obtain 3D bounding box size priors and determine the rigidity of objects, we utilized the following prompts with ChatGPT-4 [1]:\n\n- **Prompt:** Please provide the typical 3D bounding box size for *[class]*\n    \n    **Response:** Here are the typical 3D bounding box dimensions (Length × Width × Height in meters) for *[class]*, based on common datasets like nuScenes, KITTI, and the Waymo Open Dataset.\n\n- **Prompt:** Is *[class]* deformable or rigid?\n    \n    **Response:** A *[class]* is considered a deformable / rigid object.\n\nBelow is the list of categories we provided to GroundingDINO [2] as text prompts:  \n[Car, Bus, Person, Truck, Construction Vehicle, Trailer, Barrier, Bicycle, Motorcycle, Traffic Cone, Dog, Fire Hydrant, Stroller].\n\n\n## Time/Computation cost for automatic annotation by ours ##\n\nWe use 8 NVIDIA A6000 GPUs (48GB) and 2 AMD EPYC 7763 CPUs. Below is a summary of the datasets used in our experiments:\n\n- **nuScenes Dataset [3]:** approximately 150 K images and 28 K LiDAR scans\n- **Waymo Open Dataset [4]:** approximately 750 K frames images and 150K LiDAR scans\n- **Lyft Dataset [5]:** approximately 120 K images and 20 K LiDAR scans\n\n**Table 1**. Time cost for automatic annotation on various autonomous driving datasets.\n\n| | nuScenes [3] | Waymo [4] | Lyft [5] |\n|:---|---:|---:|---:|\n| Instance-level Feature Extraction | ~13 hours | ~1 day | ~10 hours |\n| Context Aware Refinement |~ 8 hours | ~10 hours | ~8 hours |\n| Adaptive 3D Bounding Box Generation | ~5 hours | ~12 hours | ~4 hours |\n\n**Table 2**. Time cost comparison with baselines. We utilize UNION [12] and CPD [6] as a baseline for nuScenes [3] and Waymo dataset [4] respectively.\n\n| | nuScenes [3] | Waymo [4] |\n|:---:|---:|---:|\n| **Ours** | ~26 hours | ~46 hours |\n| Baseline | ~46 hours | ~48 hours |\n\nTable 1 presents the time required for 3D bounding box annotation using OpenBox across different datasets. The relatively higher annotation time for the Waymo dataset [4] is attributed to its denser point clouds (64 channels) and larger dataset size. Table 2 compares the automatic annotation cost of OpenBox with other baselines. Notably, in the case of nuScenes [3], OpenBox achieves better performance despite requiring approximately 1.7× less annotation time **(please see Table 1 in the response for Reviewer GKxa)**. Similarly, for the Waymo dataset [4], although the annotation time is comparable, training with data annotated by OpenBox leads to superior detection performance (please see Table 1 in the main paper).\n\n## Training time for 3D object detection model ##\n\n- Voxel-RCNN on Waymo Open Dataset: 18 hours\n- Point RCNN on Lyft dataset: 5 hours\n- Centerpoint on nuScenes dataset: 5 hours\n\n## Clarity ##\n\nTo comprehensively evaluate the effectiveness of OpenBox, we conduct experiments under two different settings. We will include the information below in the main paper.\n\n- **Setting 1:** The training set is generated via automatic annotation, which is then used to train a 3D object detector. Evaluation is performed on a human-annotated validation set.\n\n- **Setting 2:** We directly compare automatically annotated data with human-annotated data on training set.\n\nIn the main paper, Table 1 and Table 2 correspond to Setting 1, while Table 3 and Table 4 present results from Setting 2. In addition, in Table 1, the symbol † indicates that we flip the OpenBox annotations and point clouds to obtain 360° coverage. The symbol ‡ denotes that the region outside the camera frustum is filled with annotations from CPD [6].\n\n\"Appendix\" refers to the supplementary material. Apologies for the confusion. Please refer to the supplementary material for additional information.\n\n## Other Questions ##\n\nWe appreciate the reviewer's acknowledgement regarding the efficiency of our method. We will make sure to further clarify the paper in the final version of our submission. We are open to open source the code upon acceptance of our work. The title of our approach \"OpenBox\" aims at reflecting the duality between open-vocabulary and open source. We also include a list of github repositories that helped us construct our codebase: OpenPCDet [7], CPD [6], GroundedSAM2 [8], pypatchwork++ [9]\n\n**Q: Annotation cost is pointed out in the abstract, and one of the contribution of OpenBox is that it requires no training of 3D object detectors. Typically 3D bounding box is decided by its center, size and rotation. The localization(center) of bounding box is substituted by 2D segmentation models and 3D point cloud clustering. Box size follows the typical size of the category and rotation is decided by 3D-2D IoU alignment. It is great to combine existing methods and provide amazing annotation effects. However shall we neglect the training cost of the existing methods like DINO, SAM2 and ChatGPT. Furthermore, what is the inference cost of all models. Are all the models/algorithms used in OpenBox use less resources/cost less time than a traditional 3D object detection? If possible, please provide these information in the appendix and we can have a better comparison between OpenBox and traditional 3D object detection way.**\n\nI agree to some extent with the reviewer's point that the training costs of Grounding DINO, SAM2, and ChatGPT cannot be ignored. However, all these methods are evolving into foundation models in a broader sense, serving vision tasks much like an ImageNet [10] pretrained ResNet [11] backbone. Indeed, our motivation is to leverage human level 2D visual foundation models to automate the dataset generation required for training a 3D foundation model.\n\n[1] OpenAI. (2023). GPT-4 Technical Report. arXiv.   \n[2] Liu et al. (2023). Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv.  \n[3] Caesar et al. (2020). nuscenes: A multimodal dataset for autonomous driving. In CVPR.  \n[4] Sun et al. (2020). Scalability in perception for autonomous driving: Waymo open dataset. In CVPR.  \n[5] Houston et al. (2021). One thousand and one hours: Self-driving motion prediction dataset. In Conference on Robot Learning.   \n[6] Wu et al. (2024). Commonsense prototype for outdoor unsupervised 3d object detection. In CVPR.   \n[7] OpenPCDet Development Team. (2020). Openpcdet: An open-source toolbox for 3d object detection from point clouds. Github.  \n[8] Ren et al. (2024). Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks. arXiv.  \n[9] Lee et al. (2022). Patchwork++: Fast and robust ground segmentation solving partial under-segmentation using 3D point cloud. In IROS.  \n[10] Deng et al. (2009). ImageNet: A large-scale hierarchical image database. In CVPR.  \n[11] He et al. (2015). Deep Residual Learning for Image Recognition. In CVPR.  \n[12] Lentsch et al. (2024). UNION: Unsupervised 3D Object Detection using Object Appearance-based Pseudo-Classes. In NeurIPS."}, {"review_id": "pXcFFvs5rZ", "reviewer": "Reviewer_cFQr", "summary": "The manuscript introduces a system for annotating open world 3D bounding boxes for autonomous car datasets with aligned image and point cloud data. The approach leverages 2D foundation models and aligns those outputs effectively with 3D point cloud information as well as common sense 3D object sizes. The approach seems to perform well quantitatively and qualitatively. Some of the long tail 3D bounding boxes detected by the system like strollers and dogs are impressive.", "questions": "- see weaknesses; \n    - It would also be very useful to see the baseline performance of the 3D OBB detector trained on the human annotated OBBs for both datasets. \n    - clarify if code is going to be released\n    - how is the mesh extracted from the point cloud?", "limitations": "limitations are in the main paper", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- strength:\n    - The key strength of the paper is the strong performance of the approach. The qualitative results look very good and really help drive home the quantitative results. The open world 3D OBBs like dog, stroller are great to see (although it is unclear how cherry picked they are?). The supplemental video does a great job at illustrating the performance of the method on dynamic scenes with many humans and cars and gives further evidence for the long-tail localization ability. The temporal stability of the 3D OBBs is also great to see.\n    - The manuscript has good illustrations to help communicate the system components and interplay as well as describe details like 3D bounding box extension. \n- weaknesses:\n    - The quantitative evaluation only shows comparison to methods that leverage only LiDAR data on WOD except for LiSe on Lyft. The approach's performance would be stronger supported with baselines that leverage the same modalities on WOD as well. It would also be very useful to see the baseline performance of the 3D OBB detector trained on the human annotated OBBs for tboth datasets. \n    - clarity and reproducability: The approach is a complex system of 2D foundation models, 3D clustering, 3D ground plane estimation, surface estimation, bounding box priors from ChatGPT. Even though the description in the paper is quite exhaustive and there are multiple good figures illustrating the system, I doubt that it can be reproduced well without release of the code (which is indicated for release in the paper checklist but not the main paper) \n    - clarity: the writing clarity and quality could be improved. \n        - For example, various acronyms are used without introduction which makes it hard to follow the paper. I.e. PP score (l144), CST, CBR, CPD (table 1, l218). \n        - Another example is that it is not that clearly written that all the related work shown in tables are also annotating 3D OBBs and the numbers are from using the same model to train on the annotated datasets. \n        - It is unclear how the mesh is reconstructed from the point cloud. \n\nAll in all the method seems to work really well but the complexity of the approach and some clarity issues in the writing will make it hard to reproduce. It will be important to release code to make this reproducible.", "was_revised": true, "final_justification": "I have read the rebuttal and the other reviews.\n\nThe authors have addressed my questions sufficiently. The additional baseline results with the same modalities support the contribution of the work. The eval against training straight on human GT is important to anchor the machine generated annotations. These and the interpretations should be added to the publication. I also appreciate the additional information on running time of the annotation tool as well as GPT prompts as suggested by other reviewers.\n\nTrusting that authors will add this information to the publication I will raise my rating to accept.", "rebuttal": "Thank you for the encouraging feedback. We’re glad the qualitative results and supplemental video clearly illustrate our quantitative improvements and that the temporal stability and system illustrations were helpful. The reviewer’s insights will guide us in highlighting these strengths even more effectively.\nBelow, we provide detailed responses to the reviewer’s questions. We will include additional results and discussion in the final version.\n\n## Comparison with more baselines ##\n\n**Table 1.** 3D object detection results on nuScenes [1] validation set. All methods use CenterPoint [2] . L and C denote LiDAR and camera respectively.\n\n| Method    | Modality | Car 3D AP ↑    | Pedestrian 3D AP ↑ | Cyclist 3D AP ↑ |\n|:-----------|:----------:|:----------------|:---------------------|:------------------|\n| UNION [3] | L + C    | 30.1           | 41.6                | 0.0              |\n| **OpenBox**   | L + C    | **40.9 (+10.8)**   | **62.7 (+21.1)**        | **5.2 (+5.2)**       |\n\nAs shown in Table 1, we observe performance improvements across all classes. One key reason for the significant gains is that, unlike OpenBox, UNION [3] omits the refinement process for point clouds and 3D bounding boxes. In particular, it does not explicitly consider the camera-LiDAR calibration error when projecting LiDAR point clouds on DINOv2 [4] feature maps which leads to noise at the boundary of the objects. Additionally, UNION [3] neither resizes nor re-localizes the initial 3D bounding boxes, which leads the model to predict suboptimal bounding boxes.\n\n**Table 2.** 3D object detection results on WOD [5] validation set. All values are measured under IoU=0.4  (Intersection of Union). VRU indicates vulnerable road users (pedestrian + cyclist).\n\n| Method    | Modality | Vehicle 3D AP ↑ | VRU 3D AP ↑     |\n|-----------|:----------:|:------------------|:------------------|\n| UP‑VL [6] | L + C    | 52.0             | 19.7             |\n| **OpenBox**   | L + C    |**60.7 (+8.7)**      | **22.7 (+3.0)**      |\n\nSimilarly, in Table 2, we observe a significant performance improvement compared to the previous baseline, UP-VL [6]. Although we could not verify this directly due to the lack of released code, Figure 6 (Error Analysis) in the Supplementary Material of [6] shows that classification errors (confusion with background) account for a large portion of the mistakes. We conjecture that this is because UP-VL [6] applies feature distillation directly to the point clouds without any refinement process unlike ours which employs context-aware refinement to mitigate noise introduced during 2D-to-3D unprojection.\n\n## Comparison with model trained on human annotated dataset. ##\n\n**Table 3.** 3D object detection results on WOD [5] validation set. \n\n| Annotation Method | Veh 3D AP (0.5 / 0.7) | Pedestrian 3D AP (0.3 / 0.5) ↑ | Cyclist 3D AP (0.3 / 0.5) ↑ |\n|:-------------------|:------------------------:|:-------------------------------:|:----------------------------:|\n| Human             | 93.31 / 75.70 | 87.25 / 77.93 | 58.33 / 54.88 |\n| Auto (Ours)       | 66.89 / 39.14 | 55.71 / 37.82 | 21.00 /   &nbsp;7.08 |\n\n\n**Table 4.** 3D object detection results on Lyft [7] validation set. Following [18], we evaluate in a class-agnostic manner at IoU = 0.25, and each value represents BEV AP / 3D AP.\n\n\n| Annotation Method | 0-30m         | 30-50m        | 50-80m        | 0-80m         |\n|:-------------------|:---------------:|:---------------:|:---------------:|:---------------:|\n| Human             | 82.8 / 82.6   | 70.8 / 70.3   | 50.2 / 49.6   | 69.5 / 69.1   |\n| Auto (Ours)       | 62.4 / 62.3   | 56.6 / 50.6   | 19.9 / 19.6   | 42.5 / 42.0   |\n\nTable 3 and Table 4 use [8], [9] for  3D object detector, respectively.  In Table 3, while all classes show a performance gap compared to models trained on human-annotated data, the cyclist class suffers the largest drop. This stems from limitations of 2D visual foundation models such as Grounding DINO [10]. Because [10] was not trained on the prompt “cyclist,” it cannot directly infer that category. We therefore substitute the prompt “bicycle,” which leads to inconsistent detections: in the same scene, the model may first detect both bicycle and rider, then detect only the bicycle **(please watch 01:09  in the supplementary video for a mis-localized cyclist)**. In Table 4, performance between 50 m and 80 m declines sharply. Even after aggregation, distant objects yield sparse point clouds that are then discarded during HDBSCAN [12] or subsequent refinement, leaving few annotated boxes in long range. Although our automatic annotations still underperform human labels, we expect that integrating a tracking module and dense, image-based 3D bounding box annotation [13] will help recover performance for both distant and challenging instances. Our method shows substantial progress toward human-level performance, producing training data of sufficient quality to achieve object detection results that are reasonably comparable to those trained on human-annotated datasets.\n\n## Clarity of Methodology ##\n\n**Table 5.** Method and 3D object detection network used for main paper\n\n| Method Used on Dataset        | 3D Object Detection Network |\n|------------------------------|-----------------------------|\n| WOD [5]       | Voxel RCNN [8]              |\n| └ DBSCAN [14], MODEST [15], CPD [16], OYSTER [17] |                             |\n| Lyft [7]             | Point RCNN [9]              |\n| └ LiSe [18], MODEST [15]     |                             |\n\nThough we wrote the experimental settings in line 204 - line 207 in the main paper, we will include Table 5 for improving clarity of the paper. Additionally we will include the explanation of acronyms before methodology part in the main paper for readers who are not much familiar with this area. Below is the explanation of acronyms.\n\nPP Score (Persistence Point Score) [19] : Measurement of how “persistent” a LiDAR point is across multiple passes (traversals) of the same route. Higher means high probability of static point.\n\nCBR (CProto-constrained Box Regularization for Label Refinement)  [16]: CPD [16], an unsupervised 3D object detection method, computes a quality score for each initial pseudo label based on distance, occupancy, and size similarity. The high score pseudo labels (i.e., those of highest quality) are collected into a prototype set (CProto). These prototypes are then used to refine the original pseudo labels.\n\nCST (CProto-constrained Self-Training)  [16] : [16] proposes an optimization strategy for 3D object detection networks in which both the near, dense point clouds from the CProto set and the downsampled sparse point clouds are fed as inputs. The network’s predictions on these two inputs are then constrained via a feature consistency loss and a bounding-box regression loss.  \nMesh Reconstruction : We utilize VDBFusion [20] on a sequence of point clouds for mesh reconstruction.  \n  \n## Other Questions ##\n\nWe appreciate the reviewer's acknowledgement regarding the efficiency of our method. We will make sure to further clarify the paper in the final version of our submission. We are open to open source the code upon acceptance of our work. The title of our approach \"OpenBox\" aims at reflecting the duality between open-vocabulary and open source. We also include a list of github repositories that helped us construct our codebase[16,21,22,23].\n\n[1] nuscenes: A multimodal dataset for autonomous driving. In CVPR'20  \n[2] Center-based 3D Object Detection and Tracking. In CVPR'21  \n[3] UNION: Unsupervised 3D Object Detection using Object Appearance-based Pseudo-Classes. In NeurIPS'24  \n[4] DINOv2: Learning Robust Visual Features without Supervision. arXiv'23  \n[5] Scalability in perception for autonomous driving: Waymo open dataset. In CVPR'20.  \n[6] Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving. In ICCV'23  \n[7] One thousand and one hours: Self-driving motion prediction dataset. In CoRL'21   \n[8] Voxel r-cnn: Towards high performance voxel-based 3d object detection. In AAAI'21.  \n[9] PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud. In CVPR'19.  \n[10] Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv'23.  \n[11] Sam 2: Segment anything in images and videos. arXiv'24.  \n[12] A Hybrid Approach To Hierarchical Density-based Cluster Selection. In IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, 2019 (MFI)  \n[13] Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data. In NeurIPS'24  \n[14] A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Proceedings of 2nd International Conference on Knowledge Discovery and Data Mining (1996).  \n[15] Learning to detect mobile objects from lidar scans without labels. In CVPR'22.  \n[16] Commonsense prototype for outdoor unsupervised 3d object detection. In CVPR'24.  \n[17] Oyster: Towards Unsupervised Object Detection from LiDAR Point Clouds. In CVPR'23  \n[18] Approaching Outside: Scaling Unsupervised 3D Object Detection from 2D Scene. In ECCV'24.  \n[19] Learning to detect mobile objects from lidar scans without labels. In CVPR'24.  \n[20] VDBFusion: Flexible and Efficient TSDF Integration of Range Sensor Data. Sensors. 2022  \n[21] OpenPCDet Development Team.  Openpcdet: An open-source toolbox for 3d object detection from point clouds. Github. 2020  \n[22] Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks. arXiv'24.  \n[23] Patchwork++: Fast and robust ground segmentation solving partial under-segmentation using 3D point cloud. In IROS'22"}, {"review_id": "3OCnEKwPOh", "reviewer": "Reviewer_GKxa", "summary": "This paper proposes OpenBox, a automatic 3D bounding box annotation pipeline that leverages 2D visual foundation models to generate high-quality 3D annotations for various object classes without requiring self-training. It achieves state-of-the-art performance on datasets like Waymo and Lyft by refining point clouds and adapting bounding boxes based on object physical properties.  Overall, the  paper is well-written, and the designs of 3D-2D IoU alignment and Common Statistics from ChatGPT are interesting and novel in the this filed.", "questions": "Please see Weaknesses.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n1 The paper is well-written, with clear methodology and extensive experimental results that validate its effectiveness.\n\n2 The use of 3D-2D IoU alignment and Common Statistics from ChatGPT is interesting and novel in the this filed.\n\n3 The motivation for the module design is quite clear, and the Figures are clear. For instance, by referring to Figures 2 and 3, one can quickly understand the entire process of the method and the details of the design.\n\nWeaknesses:\n\n1.\tLack of comparison with other image-involved methods.\nI noticed that the paper only compares with LiSe. However, there are several other methods that also use image data for annotation, such as LSMOL [1] and UNION [2]. A more comprehensive comparison with these approaches would strengthen the evaluation.\n\n2.\tMissing discussion on related 2D-to-3D annotation works.\nSome existing methods utilize 2D foundation models to generate 3D annotations, such as SP3D [3]. Although their problem settings may differ, they share similarities in generating 3D boxes from 2D cues. Before introducing the Context-aware Refinement module, it would be helpful to discuss how previous work has addressed the challenge of inaccurate unprojection.\n\n3.\tClarification needed on size prior from ChatGPT.\nThe size estimates obtained from ChatGPT are generally stochastic in nature—different prompts and query times may lead to varying results. Therefore, it is important to clarify the details of how the size prior was obtained, including prompt formulation and query settings.\n\n[1] Wang, Yuqi, Yuntao Chen, and Zhao-Xiang Zhang. \"4d unsupervised object discovery.\" Advances in Neural Information Processing Systems 35 (2022): 35563-35575.\n[2] Lentsch, T., Caesar, H., & Gavrila, D. (2024). Union: Unsupervised 3d object detection using object appearance-based pseudo-classes. Advances in Neural Information Processing Systems, 37, 22028-22046.\n[3] Zhao, Shijia, Qiming Xia, Xusheng Guo, Pufan Zou, Maoji Zheng, Hai Wu, Chenglu Wen, and Cheng Wang. \"SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Prompts.\" arXiv preprint arXiv:2503.06467 (2025).", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for taking the time to provide such a valuable review. We appreciate the reviewer’s recognition of our paper’s well-written presentation, clear methodology, and extensive experiments that validate its effectiveness. The novel use of 3D-2D IoU alignment and ChatGPT-derived common statistics, as well as the clearly motivated module design and illustrative figures were particularly encouraging highlights. The reviewer’s insights will guide us in further refining both the technical depth and clarity of our work.\nBelow, we provide detailed responses to the reviewer’s questions. \n\n\n**Q1. Lack of comparison with other image-involved methods. I noticed that the paper only compares with LiSe. However, there are several other methods that also use image data for annotation, such as LSMOL [1] and UNION [2]. A more comprehensive comparison with these approaches would strengthen the evaluation.**\n\n\nTable 1. 3D object detection results on nuScenes[3] validation set. All methods use CenterPoint[7] as a 3D object detector. L and C denote LiDAR and Camera respectively.\n| Method   | Modality | Car 3D AP ↑     | Pedestrian 3D AP ↑ | Cyclist 3D AP ↑ |\n|----------|----------|------------------|---------------------|------------------|\n| UNION    | L + C    | 30.1             | 41.6                | 0.0              |\n| **OpenBox (Ours)**  | L + C    | **40.9 (+10.8)**     | **62.7 (+21.1)**        |  **5.2 (+5.2)**       |\n\n\nAs shown in Table 1, we observe performance improvements across all classes. One key reason for the significant gains is that, unlike OpenBox, UNION [2] omits the refinement process for point clouds and 3D bounding boxes. In particular, it does not explicitly consider the camera-lidar calibration error when projecting LiDAR point clouds on DINOv2 [4] feature maps which leads to noise at the boundary of the objects.. Additionally, UNION [2] neither resizes nor re-localizes the initial 3D bounding boxes, which leads the model to predict suboptimal bounding boxes.\n\n                                          \nTable 2. 3D object detection results on Waymo Open Dataset [5] validation set. All values are measured under IoU = 0.4 (Intersection over Union). VRU indicates vulnerable road users (pedestrian + cyclist).\n\n| Method   | Modality | Vehicle 3D AP ↑      | VRU 3D AP ↑         |\n|----------|----------|------------------|---------------------|\n| UP-VL [6]| L + C    | 52.0             | 19.7                |\n| **OpenBox (Ours)**  | L + C    | **60.7 (+8.7)**      | **22.7 (+3.0)**         |\n\n\nSimilarly, in Table 2, we observe a significant performance improvement compared to the previous baseline, UP-VL [6]. Although we could not verify this directly due to the lack of released code, Figure 6 (Error Analysis) in the supplementary material of UP-VL [6] shows that classification errors (confusion with background) account for a large portion of the mistakes. We conjecture that this is because UP-VL [6] applies feature distillation directly to the point clouds without any refinement process—unlike OpenBox, which employs context-aware refinement to mitigate noise introduced during 2D-to-3D unprojection. Additionally, while LSMOL [1] is a relevant recent work, it does not evaluate 3D object detection and thus was not included in our experiments. However, to further promote generalizable 3D perception, we plan to evaluate our method on the 3D instance segmentation task in future work. We will include these results and discussion in the final version upon acceptance.\n\n**Q2. Missing discussion on related 2D-to-3D annotation works. Some existing methods utilize 2D foundation models to generate 3D annotations, such as SP3D [6]. Although their problem settings may differ, they share similarities in generating 3D boxes from 2D cues. Before introducing the Context-aware Refinement module, it would be helpful to discuss how previous work has addressed the challenge of inaccurate unprojection.**\n\n\nIn general, the 2D-to-3D un-projection process introduces significant noise, particularly around the boundaries of the 2D masks due to camera LiDAR calibration errors. To mitigate this, methods such as OVM3D-Det [9] and SP3D [8] attempt to reduce noise by shrinking the masks. However, these approaches still fail to address errors stemming from the imperfections of 2D visual foundation models. For example, when a vehicle is reflected on a storefront window, such reflections may be incorrectly projected into 3D space, resulting in false positives. In contrast, our context-aware refinement evaluates the ratio between clusters near reflective surfaces and the actual unprojected point cloud, allowing it to effectively remove such false positives.(Please see Table 4-(a) and Figure 4  in the main paper.)\n\n**Q3. Clarification needed on size prior from ChatGPT. The size estimates obtained from ChatGPT are generally stochastic in nature—different prompts and query times may lead to varying results. Therefore, it is important to clarify the details of how the size prior was obtained, including prompt formulation and query settings**\n\nTo obtain 3D bounding box size priors and determine the rigidity of objects, we utilized the following prompts with ChatGPT-4 [10]:\n\n**Prompt**: Please provide the typical 3D bounding box size for *[class]*\n\n**Response**: Here are the typical 3D bounding box dimensions (Length × Width × Height in meters) for *[class]* , based on common datasets like nuScenes, KITTI, and the Waymo Open Dataset.\n\n\n**Prompt**: Is *[class]* deformable or rigid?\n\n**Response**: A *[class]* is considered a deformable / rigid object.\n\n\nBelow is the list of categories we provided to Grounding DINO [11] as text prompts: \\\n[Car, Bus, Person, Truck, Construction Vehicle, Trailer, Barrier, Bicycle, Motorcycle, Traffic Cone, Dog, Fire Hydrant, Stroller].\n\n\n[1] Wang et al. (2022). 4d unsupervised object discovery. In NeurIPS. \\\n[2] Lentsch et al. (2024). UNION: Unsupervised 3D Object Detection using Object Appearance-based Pseudo-Classes. In NeurIPS. \\\n[3] Caesar et al. (2020). nuscenes: A multimodal dataset for autonomous driving. In CVPR. \\\n[4] Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision. arXiv. \\\n[5] Sun et al. (2020). Scalability in perception for autonomous driving: Waymo open dataset. In CVPR. \\\n[6] Najibi et al. (2023). Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving. In ICCV. \\\n[7] Yin et al. (2021). Center-based 3D Object Detection and Tracking. In CVPR. \\\n[8] Zhao et al. (2025). SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate\nCross-Modal Semantic Prompts. In CVPR. \\\n[9] Huang et al. (2024). Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data. In NeurIPS. \\\n[10] OpenAI. (2023). GPT-4 Technical Report. arXiv. \\\n[11] Liu et al. (2023). Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv."}]

Author rebuttal:

Meta-review: This work proposes a two-stage auto-labeling framework for annotating open world 3D bounding boxes by leveraging 2D visual foundation models. 
It received two accept and two boardline accept.
Most of the concerns have been solved during the rebuttal.
The AC recommends acceptance for this paper.
Decision: Accept (spotlight)