Just Add $100 More: Augmenting Pseudo-LiDAR Point Cloud for Resolving Class-imbalance Problem
Authors: Mincheol Chang
Siyeong Lee
Jinkyu Kim
Namil Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: Autonomous Driving
Class Imbalance
Data Augmentation
Primary area: machine_vision
TL;DR: A low-cost yet effective data augmentation framework for alleviating class imbalance in 3D object detection.

Abstract: Typical LiDAR-based 3D object detection models are trained with real-world data collection, which is often imbalanced over classes.
To deal with it, augmentation techniques are commonly used, such as copying ground truth LiDAR points and pasting them into scenes.
However, existing methods struggle with the lack of sample diversity for minority classes and the limitation of suitable placement.
In this work, we introduce a novel approach that utilizes pseudo LiDAR point clouds generated from low-cost miniatures or real-world videos, which is called Pseudo Ground Truth augmentation (PGT-Aug).
PGT-Aug involves three key steps: (i) volumetric 3D instance reconstruction using a 2D-to-3D view synthesis model, (ii) object-level domain alignment with LiDAR intensity simulation, and (iii) a hybrid context-aware placement method from ground and map information. 
We demonstrate the superiority and generality of our method through performance improvements in extensive experiments conducted on popular benchmarks, i.e., nuScenes, KITTI, and Lyft, especially for the datasets with large domain gaps captured by different LiDAR configurations.
The project webpage is https://just-add-100-more.github.io.

Just Add $100 More: Augmenting Pseudo-LiDAR Point Cloud for Resolving Class-imbaland Problem
Mincheol Changl** Korea University mlncheores@korea.co ac. kr
Siyeong Leef NAVER LABS
siyeong wew.g.geeeeeaaeeeab...oom
Jinkyu Kim
Namil Kim NAVER LABS namil. hww.llllaala...bb.boooo com
Korea University inkyukin@korea. ac .r.
Abstract
Typical LiDAR-basce 3D object detection models trained with real-world data collection, which is often imbalanced over classes. To deal with it, augmentation techniques commonly used, such copying ground truth LiDAR points and pasting them into scenes. However, existing methods struggle with the lack sample diversity for minority classes and the limitation of suitable placement. In this work, we introduce novel approach that utilizes pseudo LiDAR point clouds generated from low-cost miniatures real-world videos, which is called Pseudo Ground Truth augmentation (PGT-Aug). PGT--ug involves three key steps: (i) volumetric 3D instance reconstruction using 2D>to-3ID view synthesis model, (ii) object-level domain alignment with LiDAR intensity simulation, and (iii) a hybrid context-aware placement method from ground and map information. We demonstrate the superiority and generality of our method through performance improvements in extensive experiments conducted on popular benchmarks, i.e., nuScenes, KITTI, and Lyft, especially for the datasets with large domain gaps captured by different LiDAR configurations. The project webpage ltttts:://jutt add-100-mooe github.io
1 Introduction
LiDAR-basee 3D object detection has garnered growing interest thanks its wide applications, including fully autonomous vehicles (or robots) where the LiDAR point cloud main reli- able source for 3D scene understanding. There exists large volume literature on LiDAR- based 3D object detection models (I,77,, and they have achieved remarkable performance im- provements via better model architectures and efficient LiDAR points representation. Still, we observe that these models often suffer from the so-called class imbalance problem they tend be overfitted to frequently observed objects
Methods
Lanted
Foood H, A.A. Mi ALe
Input
Placeabiltt
Mathod
Generate
Poste
Flexbla Placosbilty
videos
Inpu Scene
Figure 1: Overview. We present PGT-Aug, novel cost-effective pipeline that generates and augments pseudo-LiDAR samples (from miniatures and web videos) to effectively reduce the performance gap between majority-class minority-class objects.
Work done during intern at NAVER LABS 'quual contribution, 'Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(of majority classes) and underfitted to rarely observed objects (of minority classes), causing a severe performance gap. Such an imbalance problem commonly observed in real-world data collections. such as KITTI g
and nuScenes
A naive solution for the imbalance problem is collecting more LiDAR data, but achieving sufficient number of long-tail samples is still practically challenging. Due to the inherent imbalance in the natural distribution, the more data we collect, the worse the imbalance. In literature, one common strategy to deal with class imbalance is a copy-and-paste-basee where objects (of minority classes) from other frames are copied into the current frame, balancing class-wise occurrences during training (see Fig. top). However, these methods are still limited in that (i) the areas where objects can be pasted depend on their original position (thus limiting the positional diversity), and (ii) sample diversity where the same object (of minority classes) can be repeatedly copied and pasted.
sample augmentation II 12
An ideal sample augmentation method for balancing performance across classes may need the 31) shape information of diverse samples (of minority classes) to enable flexible placements into a given scene. Recent Neural Radiance Field (NeRF)-based approaches [13 have demonstrated the ability to generate high-quality 3D scenes from multiple viewpoints low computational costs. Here, we advocate utilizing NeRF-based approaches to obtain the 3D shape of objects from external sources (not from other scenes within the same data). However, generating diverse real-worle 3D objects, especially of minority classes, can pose challenges due to the need for images from multiple viewpoints. To address this. we propose using miniatures and public videos to obtain 3D-rendered samples of target objects. Miniatures, commonly used for practical effects in the film industry, offer a practical solution to collect diverse samples, particularly for various types of vehicles.
In this work, we introduce Pseudo Ground Truth Augmentation (PGT-Aug), which generates point clouds of minority-class objects from two sources: (i) surround-view videos of given miniatures and (ii) public videos of real-world objects. As shown in Fig. bottom, we first utilize a 2D-to-3D renderer to reconstruct an object's 3D volumetric representation. Then, we transform it into LiDAR- like 3D point clouds by rearranging and filtering points and estimating their intensities. During training, such generated pseudo-LiDAr points are sampled and placed into appropriate places of a scene without where-to-par -pstt technical constraints. Our experiments with public datasets (i.c. nuScenes ( KITTI g and Lyft [77 demonstrate that our data augmentation with pseudo-LiDDAR points effectively improves detection performance for minority classes. Our contributions can be summarized into fhree-fold:
We introduce PGT-Aug, a novel cost-effective pipeline for LiDAR- object detectors to solve the class imbalance problem by (i) generating pseudo-LiDAR samples (from multi-view images of miniatures and public videos of an object) and (ii) augmenting them during training to balance the performance gap across classes.
Our pipeline involves novel view-agnostic pscudo-LiDAR sample generation where we reduce the domain gap between the real-world and the generated LiDAR point clouds. A series of processes, such as spatial distribution matching and data-driven intensity adjustments, achieve this.
We propose a novel map-aware augmentation technique that determines where to paste the pseudo- LiDAR samples based map-based scene context (i.e. placing an object into the appropriate locations in the given scene).
2 Related Work
LiDAR-based 3D Object Detection Datasets for Autonomous Driving. The automotive industry has shown a surge of interest in LiDAR-based 3D object detection, enhancing system reliability compared to camera-based detectors. However, creating large-scale annotations face challenges due to sensor costs, spatial distribution variations, and annotation difficulties caused by reflections and occlusions. While several annotated datasets O I7 20 exists, the limited quantity, compared the image domain, leads to both generalization issues and class imbalance 22). To mitigate this, we propose a novel approach for generating high-quality rare objects inexpensively for data augmentation in 3D object detection, addressing the class imbalance. (see Fig. in the Supplementary Material.)
Data Augmentation of LiDAR Point Clouds. Data augmentation plays a crucial role in increasing the diversity of training samples and addressing class imbalance in datasets. Two common strategies

---

Volumetric Instance Collection
Object-level Domain Alignment
Pseudo LIDAR Point Clouds
Augmentation
Ground Points (sweep)
Bird's Eye View Map
Miniatures
Web Videos
Data Collection
$ SIM
Spalial Domain Alignment
x
Reprojected Points
Segmentation
Intersilyy Domein Alignment
20 images
3D Paint
RGB-oo-iitnnll Models
Dige
PGT Bank
PGT Insortion
Poses
2D-3D
Iitertsty-lo-GGS
Rendering
Model
Ons
ReaH-UDRR
Box
PGT Box
Figure 2: Overview of Pseudo GT AuG Framework. Given multiview images, we first reconstruct their volumetric representations (Section (.T). We post-process RGB point clouds using spatial rearrangement and LiDAR intensity simulator (Section[3.2 producing pseudo-LiDAR point clouds. Such points are stored in psuedo LiDAR bank, and we paste the sampled objects into the larget scene with the proposed augmentation scheme (Section 3.3
for mitigating overfitting and enhancing model robustness are data resampling and data synthesis. Data resampling 27) primarily focus oversampling minority classes by replicating existing samples, thereby increasing the frequency of minority classes and balancing class distributions during model training. In contrast, data synthesis (28-30 relies on blending of original samples or utilizing synthetic data from generative models. This approach enhances the diversity of the training set, helping to address data scareity and improve overall model generalization.
33 for 3D renderings data augmentation [ 33) for utilizing existing data. Some works 10
Recent works have tackled 3D annotation difficulties through LiDAR simulation 10
31
32
38
attempted to reduce a domain gap between simulated and real-world data by combining them, as seen in SHIFT3D [1], which employs deep signed distance functions refine object's s shape and pose adversarially. GT-Aug I used copy-and-paste strategies, while Real-Aug [12 enhanced data utilization with realistic placement. Creating diverse. non-standard shapes requires considerable time and effort through 3D CAD modeling. On the other hand, 3D objects in the target dataset are partially observed, making free placement challenging. Different from previous approaches, collect miniatures and real-world objects to render realistic 3D samples with low cost from diverse domains, and introduce an object-level augmentation with flexible placeability.
Neural Radiance Fields and 3D Rendering, Many works have attempted to represent 3D scenes as continuous implicit representations or differentiable structures [13] 14 39 NeRF [3] rep- resented 3D geometry by approximating density and view-dependent RGB using a simple MLP architecture. Among many subsequent works 14 42 43), both Instant-NGi [5] and Plenox- els [14] contributed to faster training and rendering of 3D scenes. Instant-NGP accelerated MLPs using multilevel hash tables, while Plenoxels utilized sparse voxel grids for interpolating color and density field. These advances make the reconstruction of 3D objects casier and more precise [16]. facilitating their use in various perception tasks 444
3 Method
We introduce PGT-Aug, fast, realistic, and low-cost pipeline that generates (and collects) pseudo- LiDAR point clouds of minority-class objects from multi-view images of miniatures or real-world objects. Then, reduce the class imbalance problem, the generated point clouds are augmented into the current scene, balancing the number of occurrences across classes during training. As shown in Fig. 2 our pipeline consists of three main modules: (i) Volumetric 3D Instance Collection, where we reconstruct objects' 3D volumetric representation from multi-view images of world miniatures or objects (Section 3.T.. (ii) Object-level Domain Alignment, where we reduce the domain gap (i.e., between the generated vs. the collected from LiDAR sensors) by transforming the point clouds based on sensor configurations and intensity simulation models (Section 3.2). Lastly, (iii) Pseudo LiDAR Point Clouds Augmentation, where augment the generated pseudo-LiDAR

---

point clouds into the current scene by determining more realistic object insertion areas based on the estimated ground areas and map information (Section 3.3
3.1 Volumetric 3D Instance Collection of Minority Class Objects
Data Collection. Conventional Ground truth (GT) sampling-based data augmentation approaches resample minority-class 3D instances (e.g., trucks, trailers, buses, construction vehicles) from in- domain collections, but their placement ficxibility and contextual diversity are often limited. To resolve this issue, we want to augment minority-class 3D instances from out-of-domain sources, efficiently resolving the data imbalance problem by sampling more diverse types and shapes of minority-class samples in the wild. To this end, advocate for utilizing the following two cost- effective sources: (i) videos capturing the surround view of miniatures and (ii) publicly available videos capturing multiple views of minority
objects (see Fig. 2). As the title suggests, purchased dozens of realistic miniatures for just $100. Plus, we collect public videos from YouTube, followed by manual filtering to ensure those videos capture an object from multiple viewpoints.
Preprocessing. Given collected video frames, we first estimate the following three pieces of information as a preprocessing for the later 2D-to-3D rendering: (i) a camera intrinsic matrix, (ii) camera poses (3D camera position and its orientation for each frame), and (iii) binary masks for a foreground object. For (i) and (ii), we use COLMAP 45), similar to the preprocess of existing NeRF-based approaches 13
14 46 For (iii), we use the off-the-shelf video segmentation model, ie.. Segment and Track Anything
to extract segmentation masks for foreground objects.
2D-to-3D Rendering. Our framework is built on advanced 2D-to-3D renderers, such Plenoxels [14 Soussian-Splaating
and Gaussian- Splatting [16], known for efficient, high-quality 3D reconstruction. Unlike some 3D conditional generative models for specific classes such as Shap-E [48] and Zero-1-to-3 [49], these methods offer dense 3D point clouds suitable for increasing out-o--distribution samples. In general, these methods [14 [6] predict view-dependent representation, but obtaining fully visible and uniformly high-density data from all viewpoints is necessary to generate pseudo- objects and axis-aligned bounding boxes. Therefore, we propose an ad-hoc module obtain representative colors for each voxel grid or point based on the estimated mean color values from different views. More specifically, iterating through N views, it determines which voxel grid or point's camera ray passes through and calculates the color value of voxels or points by doing a dot product between corresponding spherical harmonic coefficients and view-dependent harmonies basis. If certain voxel or a point is hit multiple times by rays from different views, we update the existing value by adding the new color value and store the total number of passes. Notably, this straightforward module is applicable not only to 2D-to-3D rendering techniques such as Plenoxels and Gaussian Splatting, but also renderers that do not rely on spherical harmonics"
3.2 Object-level Domain Alignment
We aim augment real LiDAR data with simulated minority-class samples derived from RGB-colored point clouds. However, these point clouds lack crucial information regarding spatial distribution from sensor configurations and LiDAR''s intensity, which is essential for accurate LiDAR modeling and evaluation. Thus, we propose object-level domain alignment between RGB-colored point clouds Dryb and real-world point clouds D Pint. employing two alignment techniques (i) view-dependent points filtering and rearrangement and (ii) LiDAR intensity simulation.
View-dependent Points Filtering. This step simulates realistic data variations based on the object's relative position to the LiDAR sensors and their settings. Therefore, we re- quire alignment parameters based on factors such as type, position, quantity, and specifica- tions (Field of View and azimuth resolution) of LiDAR sensors in the target dataset. We ap- ply the following three steps: (i) transformation of points to the range view representation. (ii) filtering of points on the invisible side, and (iii) reprojection of points into 3D space.
IIf a renderer with spherical harmonic lighting is selected, the Oth coefficient of the spherical harmonics can be directly used to extract view-agne agnostic colors. However, we empirically observed that our approach is more robust capturing the original object's color and areas of dark shades (see Table 13 in Appendix).

---

In (i), given points in the spherical co- ordinates system, we map points into range view R(u, v) E #R where u and e are the spatial grid indexes. For each grid (u, v), point with mini- mum depth remains, and the others are filtered out (i.e., points on visible parts remain). The remaining points are ir- regularly located in 3D space, gener- ally different from those of real-world LiDAR points. We therefore rearrange each point to be regularly spaced by adjusting their inclination e and azimuth 0 in the spherical coordinate system: ø' = (=+0.5) FOV FOV down; O' = [2 (=+ - 1]. where FOV down and FOV represent the down part and the total range of the field of view, respectively. Rearranged inclination and azimuth d', 0' are calculated by backprojecting image coordinates u, u spherical coordinates following the above equations. The results on different LiDAR settings are shown in Fig. 3
32 beam
64
30-110"
25
25h-15
1080
1024
1800
RGB Point Clouds
nuSoenes
KITTI
Lyft
Figure 3: Effect of Reprojection Different Datasets.
H
LiDAR Intensity Simulation. Since object surface reflectivity cannot be derived from images, we use a data-driven model for intensity estimation. As no real-world LiDAR points match generated RGB points, we create an unpaired domain transfer model specifically for point clouds based on CycleGAN 50 framework. Also, we design novel region matching loss that directly reduces the intensity error between two samples, even with varying point counts, as shown in Fig. To compute the loss, both generated points GD pa- (Prgh) and real-world points Pint are grouped into an equal number of ball patches whose centers are obtained via farthest point sampling. After dividing into N ball patches, Hungarian matching is used to find the optimal assignment of ball patch pairs by computing the center distances for all matchings GN.
Ô = arg min CE6Y
Σ lle -C(t)ll1
(I)
where G2 and Co(s) are centers of ball patches from EDD-ga+DDun (Prgb) and Pint. respectively. After finding optimal pairs, we reduce all intensity distances between fake and real pairs of patches. For normalized points æ. y and given ball radius r,
Ball Patch Matching
Egroup Σ EEED (x) - Eyer (v)ll,,
(2)
where Exer (1), Eyetges (v) denote the average of intensity values of optimal ball patch pairs bj. (a and is 0. The overall objective function is
G Diversstr (Prge)
Proceusity
Figure 4: Region Matching Loss.
Ltotal = LcydeGAN + X.
(3)
where À is a regularizing factor set to 0.1.
Instance Size Setting. We analyzed the average object size per class in the dataset. We determine the size by adding Gaussian noise with a of 0.1. The noise exceeding one a range was clipped to a.
3.3 Pseudo LiDAR Point Clouds Augmentation
Ground and Map Synthesis for Object Insertion. To determine feasible insertion areas, Real- Aug [12 and Lidar-Aug [10] utilized estimated ground estimation. However, relying solely on this data may not cover areas adequately as shown in Fig. [5] To address this, we propose method that combines map information and estimated ground areas for more realistic scene composition. We first create a rasterized map with a 0. 128m per pixel resolution within a radius of 51.2m around the ego vehicle, assigning proper layouts (e.g., Road, Sidewalk, etc.) for inserted objects. Additionally, we construct a rasterized ground map based on estimated ground points. The two pieces of information may overlap at pixels where dynamic object are present or where ground area is not estimated due to
5

---

Inserted
Misplaced Samples
GT Box
(a) Ground-only Composition
(b) Ground+Map Composition
Figure Comparison Ground-only and Ground+Map Scene Composition. Blue and Pink-colored points denote the feasible location of insertion derived from (a) ground-only and (b) mround+map synthesized insertions, respectively.
insufficient points. To deal with it, we prioritize map values for pixels with low point density, while using ground values otherwise. As shown in Fig. (b), the proposed method can predict broader and feasible areas for more realistic data augmentation than ground-only composition.
Aligning to Data Geometry. The generated objects need be axis-aligned to easily obtain accurate bounding box annotations for the detection task. For this, we first use PCA to align the generated points and rotate them around the normal direction in sy. yz, and planes to align its axes. We also need to classify objects' front and back for heading information, so we adopt PointNet++ (51] as heading classifier trained using binary cross-entropy loss. Since PointNet+ rotation sensitive (52) and is trained without rotation augmentation, the model can distinguish between front and back.
Virtual Object Sweeps. LiDAR scan sweeping commonly used to increase the point density in some popular benchmarks 8 19. Since our pipeline can generate objects that appear from single scan, additional techniques are required apply them such datasets. We employ rigid body motion model to stack points over time. Given the dataset, we first collect the velocity and acceleration of each class's i center points at each time step and estimate motion trajectory. We translate the center of generated object points along the selected trajectory. Ultimately, we can generate virtual objects that closely resemble what is acquired from the real LiDAR sweeps.
4 Experiments
Implementation Details. We use Plenoxels 114 to reconstruct given object's 3D shape from multi- view images. Also, we utilize PointNeXt (53) in our CycleGAN-based LiDAR intensity simulator. Note that we sample 300 instances (that has at least 256 LiDAR points) for each minority class from the nuScenes dataset 9 to train our intensity simulator. We implement and evaluate our proposed
+10m
Volumetric Rendering
+5m
Distance
-5m
-15m
Object Coordinate
-150
-120
-60
-30
30
60
90
120
150"
180"
Object Heading
Figure 6: Examples generated LsDUdo-LiDAR point samples with different orientations and ranges given reconstructed 3D volumetric representations.

---

Table 1: Detection performance comparison on nuScenes val set in terms of AP, mAP, and NDS. Based on CV- Voxel [5], we compare different placement methods, such as random [1 ground- based 12]. and our placement. We also report the effect of using our pseudo-LiDAR samples.
GT-Aug ① (Random Placement)
/
x
X
x
X
Real-Aug [2] (Ground- based Placement)
X
/
/
x
X
Ours (Ground++Map-based Placement)
X
X
/
/
w/ Pseudo-LiDDR Samples
X
/
X
x
/
Class
objects vol
Performance
Bus
3,009
69.1
71.3 (12.2%)
71.8
71.7 (4.11%
73.0
72.1
(-0.9%)
Construction Vehicle
2,387
18.0
18.1 (+0.1%)
22.7
24.0 (-1.3%
23.7
24.2
(+0.5%)
Trailer
3,765
36.1
35.7 (-0.1%)
41.6
41.4 (40.2%)
42.3
40.4 (-1.%)
Truck
13,950
56.9
57.8 (40.9%)
57.3
58.3 (-1.09)
58.8
59.8 (+1.0%)
Motorcycle
2,227
60.3
61.5 (+1.25%)
67.1
68.0 (40.%)
68.7
70.3 (41.68)
Bicycle
2,071
41.5
45.1 (+3.6%)
54.9
57.1 (+2.2%)
56.7
58.3 (+1.6%)
mAP (for all 10 classes)
59.0
59.8 (+0.8%)
62.3
63.0 (+0.7%)
63.3
63.5 (+0.28)
NDS (for all 10 classes)
66.5
66.9 (40.4%)
68.4
68.7 (40.3%)
68.7
69.1 (40.4%)
method based on LiDAR-based 3D object detection implementations from OpenPCDet [54] with default parameter settings. All detectors are trained with batch size of 32 on x A100 GPUs for 20 epochs. We conducted all experiments with the fixed seed for a fair comparison. Details on architectures and hyperparameters
be found in the Supplementary Material.
Pseudo Object Bank Details. To validate our system, we have chosen nuScenes ( dataset. For the pseudo object bank, we generated minority class objects in 13 heading directions (from - 180 to 180% at intervals of 30") within the entire perception range (from -50m to 50m at intervals of 5 meters). During training, we discard objects with less than 16 points to prevent ambiguity between classes and add a small variation at the center. As a result, our bank has 36,960 trucks, 52,800 construction vehicles, 12,960 buses, 19,280 trailers, 25,279 motorcycles, and 4,300 bicycles.
Performance on NuScenes val set. We assess the effect of adding pseudo-LiDAR samples during training using Co-Voxel [5] with a voxel size of [0.075, 0.075, 0.2]. As shown Table 1 the incorporation of pseudo-LiDAR samples into the GT database significantly improves the overall detection performance, including minority classes, terms of mAP and NDS (compare 1 vs. 2nd, 3rd vs. 4th, and 5th vs. 6th columns). Our placement approach surpasses existing methods (compare 1st, 3rd vs. 5th columns). Combining our two methods achieves the best performance, with 69. 19.1% in NDS and 63.5% in mAP, suggesting that our high-quality pscudo-LiDAR samples effectively improve detection performance for minority classes without compromising majority classes.
Performance on NuScenes test set. Lastly, we evaluate model nuScenes test set based on CP-Voxel SS and Transfusion-L 55 as baseline detection models. We observe in Table 2 that our model consistently achieves the best NDS and mAP scores (compared to Real-Aug (22). Notably, our method shows significant performance gains for minor classes such as Trailer, Truck, Motorcycle, etc. Note that we applied the same Test Time Augmentation (TTA) technique Co-Voxel as their official submission to nuScenes leaderboard.
Table Defection performance comparison on nuScenes test set in terms of minority class AP, mAP for all 10 classes, and NDS. t; our reproduction, 3: test time augmentation enabled.
Minority classes
Model
Aug.
mAP
NDS
Bus
C.V
Trailer
Truck
M.C
B.C
GT-Aug
64.4
31.0
60.0
47.2
65.7
41.0
63.8
68.7
CP- Voxel S
Real-Aug!
64.5
29.0
60.1
57.3
72.2
47.1
65.8
71.3
PGT-Aug
68.1
29.0
61.7
57.7
74,0
48.6
67.1 (61.3)
72.3 (41.00)
GT-Aug
63.7
29.0
58.7
46.3
67.1
44.2
63.9
68.6
Transfusion-L
55
Real-Aug
64.3
31.0
60.0
47.3
65.7
41.0
63.8
68.7
PGT-Aug
67.3
30.1
60.2
56,9
68.2
40.6
65.1 (+1.3%)
69.9 (+1.2%)
7

---

Table 3: Comparisons between baseline and PGT-Aug for individual models nuScenes val set. Abbr. C.V: Construction Vehicle, Ped: Pedestrian, T.C. Traffic Cone, M.C. Motorcycle, B.C. Bicycle.
:: our reproduction.
Majority classes
Minority
Model
Aug.
Car
Ped
Barrier
T.C
Bus
C.V
Trailer
Truck
M.C
B.C
MAP
NDS
GTT-Aug
81.5
77.3
58.3
57.9
67.2
15.2
36.2
50.6
40.7
16.2
50.11
61.56
SECOND 1
Real-Aug!
59.09
67.23
PGT-Aug
84.8
80.0
62.2
67.2
71.9
24.4
42.4
58.8
64.0
38.1
59.37
67.30
GT-Aug
83.1
82.5
65.0
65.8
63.0
14.1
23.7
54.4
51.4
25.5
52.85
62.57
CP-Pillar
Real-Aug!
82.9
84.0
65.7
67,9
64.9
19.5
26.3
57.2
64.6
46.2
58.03
64.85
PGT-Aug
83.3
83.5
65.7
67.4
22.1
27.5
65.0
58.39
65.49
GT-Aug
84.9
85.4
68.3
69.9
69.1
18.0
36.1
56.9
60.3
41.5
59.04
66.54
CP-Voxel 3
Real-Aug!
84.7
85.0
67.5
70.0
71.8
22.7
41.6
57.3
67.1
54.9
62.27
68.39
PGT-Aug
85.4
85.4
68.0
71.1
72.1
70.3
63.52
69.11
GT-Aug
86.6
73.6
72.7
56.1
63.48
68.58
Transfusion-L
55
Real-Aug!
86.6
86.9
69.9
73.6
73.1
25.9
42.2
53.8
68.3
55.1
63.54
68.57
PGT--ug
87.0
86.4
69.4
74.0
73.4
26.7
46.9
50.6
71.3
56.3
64.20
68.83
GT-Aug
83.7
84.5
68.9
68.4
71.4
20.9
37.5
56.
62.9
49.8
60.42
67.03
VoxeINeXt
Real-Aug
67.8
62.25
67.62
PGT-Aug
84.2
85.0
67.7
70.6
72.4
23.9
40.6
57.5
70.9
56.6
62.95
68.30
Comparison with Different Model Architectures. We compared detection performance among GT-Aug, Real-Aug, and PGT-Aug using five baselines, detailed in Table All detectors were tested under same conditions, ensuring that no additional information about the generated objects was utilized during the evaluation process. PGT-Aug brings significant improvements over other augmentations in all types of models, such as center-based models (CP-Pillar, CP-Voxel [5]), anchor- based model (SECOND M., and other types (VoxelNeXt (6) Transfusion-L (55)). This confirms that our method can generally be applied to various detection models, boosting the model's accuracy minority classes without sacrificing accuracy for majority classes.
Ablation Studies
Quality of Pseudo Labels. To evaluate the quality of our generated objects, we measure FID scores using SE(3) -taesf)tranf [56] trained on the nuScenes dataset. As shown in Table our generated Beeuo-LLiDA objects show similar lower FID scores (than true samples), which may confirm their plausibility and high quality (compare 5th and 6th vs. 7th and 8th columns). We also observe that the pseudo-LiDAR object quality improves with (i) similar Azimuth resolution to the target dataset sensor, (ii) the use of RGB values additionally as input, and (iii) the regularization by group
Table 4: Quality Pseudo LiDAR Point Clouds. FID scores (squared Wasserstein distance between given samples and nuScenes samples, thus lower is better) comparison between variants of our models and public LiDAR datasets, Lyft [17 and A2D2 [8 Abbr. G.S: Gaussian Splatting
Pseudo-I - LiDAR Point Clouds
Lyft 17
A2D2
18
Volumetric 3D type
Plenoxels [14
G.S [16
Azimuth Resolution (px)
3600
1080
1080
1080
1080
1080
RGB features
Group intensity Loss
/
X
Bus
17.7
14.6
13.1
13.0
13.2
11.2
8.7
19.8
Construction Vehicle
7.0
7.5
7.6
7.5
7.6
7.6
6.0
Trailer
20.6
12.7
11.9
11.9
12.2
13.7
36.5
Truck
8.9
8.4
7.6
7.6
7.3
6.9
6.6
13.4
Motorcycle
20.7
2.5
7.0
7.2
3.7
1.3
3.0
10.1
Bicycle
9.0
3.3
2.2
2.4
2.1
1.8
1.8
0.7
Avg. FID Score
14.2
8.2
8.3
8.3
7.7
7.1
4.8
14.4
mAP (for all 10 classes)
63.40
63.44
63.48
63.41
63.52
63.77
63,45
63.17
NDS (for all 10 classes)
68.83
68.99
69.02
68.87
69.11
69.35
68.88
68.73

---

Table 5: Mixing rafio between GT and PGT objects.
Table 6: LiDAR data.
Continuously increasing pseudo-
GT:PGT
0:1
1:3
1:1
3:1
1:0
Size
bank 1/4
bank 1/2
Full bank
mAP (*)
61.29
63.35
63.52
63.40
63.34
mAP (t)
63. 13.18
63.52 68.94
63,43
NDS (f)
67.63
69.08
69.11
68.78
68.71
NDS (1)
68.72
69.11
Table 7: Comparison between other methods that aim for class imbalance problems. Abbr, C.V: Construction Vehicle, Ped: Pedestrian, T.C: Traffic Cone, M.C: Motorcycle, B.C: Bicycle. i: our reproduction.
Majority classes
Minority classes
Model
Aug.
Car
Ped
Barrier
T.C
Bus
C.V
Trailer
Truck
M.C.
B.C
mAP
CBLoss
82.7
74.7
54.0
52.1
51.2
61.7
17.9
30.5
48.7
20.5
49.4
PointPillars
g
DWA
81.0
72.3
50.2
50.1
49.0
63.4
10.7
34.3
32.9
6.9
44.6
PGT-Aug
83.0
71.8
54.8
69.7
20.2
39.5
PGT Aug CBLuss
82.7
74.7
56.5
55.9
54.4
68.7
20.9
34,1
53.5
20.5
52.2 (+1.9)
intensity loss. Finally, we used objects generated by Plenoxel with the parameters that gave the best FID score for all 3D object detection tasks. We provide more details in the Supplementary Material.
Performance based on FID scores. We conducted experiments to assess how the quality of generated objects as FID scores affects detection performance. We generated objects in all comparison groups to have the same shape distribution. As shown in the last three rows of columns to of Table we observe positive correlation between the average FID scores and the detection performance.
The evaluation was conducted not only through self-validation but also using other public datasets such Lyft [17 and A2D2 [18]. Despite differences in class ontology and sensor settings from nuScenes, we can prove the effectiveness of our augmentation and confirm correlation between the FID score and the number of samples in performance gain as shown in Table See the Supplementary Material for matching classes across datasets.
Samples from Other Datasets.
Samples from Different Renderer. To demonstrate the baseline capability the proposed pipeline, we applied the results generated by replacing the rendering model with Gaussian-splattin [16] instead of Plenoxels [4]. Due to Gaussian-splatting [16] generates higher-quality objects, so the detection performance improves accordingly, as shown in Table
Mixing Ratio for Domain Alignments. Even when several methods are applied reduce the domain gap between the original and generated pseudo objects, there is still a difference between the two sample groups. Therefore, when out-of-distribution data are only used, we can see that the detection performance might drop. To alleviate this discrepancy, we used a strategy that combines the original object bank with the pseudo object bank to reduce the gap. We experimentally demonstrate that this strategy can mitigate the domain gap, allowing pseudo objects to be used effectively for detection tasks. is especially effective when the mixing ratio is 1:1, which can be confirmed through Table S
Performance based Bank Size. We discuss the importance of the size of the object bank in the detection task. To examine the impact of varying object shapes, we adjusted the size of the generated bank by decreasing the number of object shapes per class by 1/2 and 1/4, As shown in Table 6 found that as the types of object increased, the performance improved in the validation set.
Comparison between other approaches for class imbalance problems. As shown in Table 7 we provide the results of comparative experiments with 34 57]. To align with the baseline of Dynamic Weight Average (DWA) (34. we conducted experiments on PointPillars [3]. We re-implemented Class-Balann GlasneBl Loss (CBLoss) 57 with a beta value of 0.9999 and resampled the number of objects.
While we observed that loss-balancing methods like DWA 34 and CBLoss [57] effectively improve the performance on minority classes, PGT-Aug, a data-augmeentaiion-baa largest performance improvement among all methods. Furthermore, our experiments demonstrate that combining these two types of methods can yield synergistic effect.
method, achieved the

---

Table 8: PGT Performance KITTI val set in terms of AP and mAP.
Target classes
Other classes
Class
Cyelist
Car
Podestrian
Model
mAP
# objects in wal
290
262
56
2980
5082
3116
1139
605
Difficulty
Easy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
SECOND 1
GT-Aug PGT-Aug
62.3
49.8
91.4
82.4
79.6
87.1
67.9
63.3
56.2
50.2
90.7
82.1
79.3
90.3
72.1
67.7
70.1
Table 9: PGT Performance on Lyft val set. Abbr. E.V: Emergency Vehicle, O.V: Other Vehicle, M.C. Motorcycle, B.C. Bicycle. Ped: Pedestrian
Target classes
Other classes
Class
Model
Truck
Bus
O.V
M.C
B.C
Car
Ped.
mAP
# of objects in val
2,721
1,653
4,920
187
3,347
91529
4952
GT-Aug PGT-Aug
19.15
20.48
31.91
4.54
5.31
37.14
6.00
13.84
CP-Voxel S
19.85
21.11
31.99
4.39
5.48
37.11
6.12
14.01
(+0.17%)
Performance on Other Datasets. Lastly, we applied the proposed framework to KITTI and Lyft 17 with different sensor configurations, ranges, and the number of sweeps. Both datasets did not provide map information, so we modified our scene composition similar to the ground-only composition by using Patchworks [58] to estimate ground. As shown in Table 8 and Table we can see that our framework demonstrates performance improvement across datasets and applicable in various environments.
6 Conclusion, Limitations, and Broader Impacts
In this paper, we propose PGT-Aug, a low-cost yet effective data augmentation framework for class imbalance in 3D) object detection. To efficiently obtain rare class objects, start by generating objects from miniatures or web videos at low cost, then transform them to resemble real LiDAR data, and finally apply map-assistant data augmentation to insert them consistently into the scene information. Even though PGT-Aug achieves significant improvements on various 3D object detec- tion benchmarks, we believe that domain discrepancies still exist in both datasets and categories. Furthermore, our approach must be extended to dynamic and deformable objects, such as animals. While the generated pseudo objects have the potential render both images and point clouds, our current implementation is limited to point clouds. Regarding broader impact, our work applies not only objects for autonomous driving but also secure facilities or military equipment in the same data format. Thus, it could be used for the development of Al models in military or security applications.
10

---

References
[1] Yan Yan, Yuxing Mao, and Bo Li. SECOND: spansely embedded convolutional detection. Sensors, 18(10);
3337, 2018. doi: 10.3390/S18103337, URL htttp::////i.oortet org/ /00 3390/ 00990//1810337
[2] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In 2018 IEEF Conference Computer Vision and Puttern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pages 4490-4499, Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1100/CVPR.20188. 200.00077..
URL httt:::/ thecvf , com/ content_cvptttttecceeteopppp 2018/html/2hou_oore Voxeeoou__veeetttttteeee End to- End Leadd_Leeannn__eeeee 2018PPPPR_2_11___1__ html
[3] Alex H. Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 12697-12705. Computer Vision Foundation / TEEE, 2019. doi: 10. 10.1109/CVPR22019.000 URL http:///penaccesss. thecvf com/ Batnm/oontent_cVPPPP CV pacoders_for.e http://ctDDeeeciiooooooooooooo cttpee_ccddd___cc______ CVPR_ SWVPR_0099__000 2019_paper html
[4] Shaoshuai Shi, Chaoxu Guo, Li Jiang. Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li.
PV-RCNN: poxnt-voxel feature set abstraction for 3d object detection. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seatle, WA, USA, June 13-119 2020, pages 10526-10535. Computer Vision Foundation IEFF, 2020. doi: 10. W0//C//2200000000000
URL
https://openacceet thecvf. com www.oo/.///ccccc.... http//ccteettt.cccm CVPR_ 2020/html http0//htmm///h/..e/mm LU-PPV-RCC PV-R Point- Voxel_ "eature_Bet_ ott.://eoooo..........eeeeeeetar 2otp::o0___..P.222__P___Pee papppppppppp.... html
[5] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühh. Center-based 3d object detection and tracking. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25,
2021, pages 11784-11793. Computer Vision Foundation IEEE, 2021. doi: 10.1 109/CVPR46437.202 01161. URL. httptt::pppppaeeeeeeeeeeeee aD_Object_Detecttti and Twwcking.
thecvf. wwtpcccc.cocce//ooo celoo//onnent//V/ (ttpeeeet///v/2222/////a h/iii///mmmi//innnmmmemm Center-
paper html
[6] Yukang Chen, Jianhui Jhen,Jianhui
Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17 24, 2023, pages 21674-21683. IEEE, 2023. doi: 10.1109/ BtRL/htps:///eet org/ 1109//VPR52229 2023. 2023.02076
CVPR52729.22 0111229.222.000777
[7] Haiyang Wang. Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang. Di He. Bernt Schicle, and Liwei Wang. DSVT: dynamic sparse voxel transformer with rotated sets. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver; BC, Canada, June 17-24, 2023, pages 13520-13529, IEEE 2023, doi: 10.1109/VVRR5322922222. 20..3001999......99 httpa. //doi. org/ 10. 1199/CVP55222 2023.01299
[8] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. Int. J. Robotics Res., 32(11))1211-1237 2013. doi: 10. 1111770078336499149999999999 URL https
7/doi.org aaboi.org/100 10. 1171 11777002783699114919999999999999
[9] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan. and Oscar Beijbom. nuscenes: A mul- timodal dataset for autonomous driving. In 2020 IEEE/CVF Conference on Computer Vi- sion and Puttern Recognition, CVPR 2020. Seattle, WA. USA, June 13-19, 2020, pages 11618-11633 11628. Computer Vision Foundation / IEEE, 2020. doi: 10. N0.1099/CVPR4260022000 01164. URL htttps://openaceeeoo thecff. com/ content, CVPR_ MMMMullmooa hatce/mmmaaaaaatate..emo for Autonomous_ Swwtiiouuuuurviiiii wwwiviigggoio wwwpg/nng..___._eeeeeee 1020_paper..._
html html
[10] Jin Fang, Xinxin Zuo, Dingfu Zhou, Shengze Jin, Sen Wang, and I Liangjun Zhang. Lidar-aug: A general rendering-based augmentation framework for 3d object detection. In IEEE Conference on Computer Vision and Patiern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 4710-4720. Computer Vision Foun- dation IEEE, 2021. doi: 10.1 1w..109//VP444.....00 200.44377222221 .00468. URL/jattss://ppaaceee Dontent//VPR2001/httmmma/ Framevork_for: 3D httpeooorr__ooooooo_eee_______
thecvf . com/ LiDAR Aug_A A_ General Let.eerrr..eddderigggaaaeeaaeear Based
CVPR_ 2021 20____2peer__._.._ html
[11] Hongge Chen, Zhao Chen, Gregory P. Meyer, Dennis Park, Carl Vondrick, Ashish Shrivastava, and Yuning Chai. SHIFT3D: synthesizing hard inputs for tricking 3d detectors. CoRR, abs/2309. 008000, 2023. doi: 1044850/ARXXI.23300055
10. URL https::/ org/ 48550/ arriv 2309. 05810
[12] Jinglin Zhan, Tiejun Liu, Rengang Li. Jingwei Zhang. Zhaoxiang Zhang. and Yuntao Chen. Real-aug: Realistic scene synthesis for lidar augmentation in 3d object detection. CoRR, abs/2305. 128533, 2023. doi: 10. 00.48550/ARII..300.. 12855. 12853.
URL BRLpsttps: pattts://daue //ooi. org/10 48550/ ariiv. 2305. 12853
11

---

[13] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: representing scenes neural radiance fields for view synthesis. Commun. ACM. 63(1)999-106
2022. doi: 10.1145/3503250.
URL/ttps:: //doi. /rgpp//////.....././/..////// 1ete///0.114/355000 1145/3503200
[14] Sara Fridovich-Keil Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5491-5500. IEEE, 2022. 1109/ (009///PR52699 2022.00543
doi: 10.1 10.110999VR5228..00000 202..00052 URL hattts:////i....
[15] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. Graph, 10::1:00021102:115 2022. doi: 10.1145/ 3528223.3530127. URL URL https://doi. /ttp:::://ooo.or/1//// 21111101111//32222233333333
[16] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans, Graph. 49:4::19911119:11, 2023, doi: 10.1 145/55555594333
URL httpttttt/ooooooooo org/ 10. 0144/3592433
[17]
John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska. One thousand and one hours: Self-driving motion prediction dataset. In Jens Kober, Fabio Ramos, and Claire J. Tomlin, editors, 4th Conference Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pages 409 418, PMLR, 2020. http:ttts:////ooeeeee #155/houston21e html
mml m...s.sss////ss/////
[18] Jakob Geyer, Yohannes Kassahun, Mentar Mahmudi, Xavier Ricou, Rupesh Durgesh, Andrew S. Chung, Lorenz Hauswald, Viet Hoang Pham, Maximilian Mühlegg, Sebastian Dorn, Tiffany Fernandez, Martin Jänicke. Sudesh Mirashi, Chiragkumar Savani, Martin Sturm, Oleksandr Vorobiov. Martin Oelker, Sebastian Garreis, and Peter Schuberth. A2D2: audi autonomous driving dataset. CoRR, abs/2004.063200, 2020. URL https://axx:iiii org/ absa 200./2004.0000 06320
[19] Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chao- giang Yc, Wei Zhang, Zhenguo Li, Jie Yu, Chunjing Xu, and Hang Xu. One million scenes for autonomous driving: ONCE dataset. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
http:///ddaaaattteeeeeeeae proceeenaaaaeeeeeeeeeeinng prteepeingee.eeeeeeeeee neurips round1 html
cc/ pc/paper/202////aee//p/ http://2222hhhaaa/aaaaa 0777667877.555555554444888866613
[20] Jieru Mei, Alex Zihao Zhu, Xinchen Yan, Hang Yan. Siyuan Qiao, Liang-Chieh Chen, and Henrik Kret- zschmar. Waymo open dataset: Panoramic video panoptic segmentation. In Shai Avidan, Gahriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, volume 13689 of Lecture Notes in Computer Science, pages 72- Springer, 2022. doi: 1-0007997833031-111 19818-224444 JRL/httpa:// http::/ddoooooooo../ 1055//00.007//911 3-1 -718-88------1-11------11------- 19818-244
[21] Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. Imbalance problems in object detection: A review. IEEE Trans. Pattern Anal. Mach. Intell., 43(1))33388 3415, 2021. doi: 10. 22.0 2981890. URL https:///oi. wat...rg///o 10. 1109/TPAMI. 2020. 2981890
[22] Lanxiao Li and Michael Heizmann, Applying plain transformers to real-world point clouds. CoRR, abs/2303.000m 0008000006000088 2023. doi: 10.445500AAXXIV2233.00000 URL. htttp::///aat org/10 4855448555/3.0
2303. 00086
[23] Hongyu Guo and Herna Viktor. Learning from imbalanced data sets with boosting and data generation: the databoost-im approach. SIGKDD Explor. 6(1):30-39, 2004. doi: 10.1145/10077300000777 URL
https://doi. org/10. 10. 1145/ 00////0077300. 1007736
[24] Nitesh Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: synthetic minority sampligg technique. J. Artif. Intell. Res., 16:321-357, 2002, doi: 10. 10.1613/31RR.95 URL
https://doi. org/ 10. 1613/ jat./jair..... .533
[25] Xinting Hu, Yi Jiang, Kaihua Tang. Jingyuan Chen, Chunyan Miao, and Hanwang Zhang. Learning segment the tail. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2020, Seattle, WA, USA. June 13-19, 2020, pages 14042 14051. Computer Vision Foundation / IEEE, 2020. doi: 10. WWW.CC/VRRR0002220.0000
URL https: h/tp:///c/s//cccccsssee
wwrreeessssttccceeeeeee thecvf. com/ content..tttttccoc Srtpitteet_cc_______ CVPR_2020/
html Html/Hu,Lee LeaffluLeerrinn__eee SeretttoSSeeeeeeeeeeeeeeeeeeeeee the Tache_Tail__VVV _VPp_ 2e_____VR___________
_awperppppppr... .html
12

---

[26]
Shin Ando and Chun- Huang. Deep over-sampling framework for classifying imbalanced data. In Michelangelo Ceci, Jaakko Hollmén, Ljupco Todorovski, Celine Vens, and Saso Dzeroski, editors. Machine Learning and Knowledge Discovery in Dutabuses European Conference, ECML PKDD 2017. Skopje, Macedonia, September 18-22, 2017, Proceedings, Part I, volume 10534 of Lecture Notes in Computer Science, pages 770- 785. Springer, 2017. doi:
3-17997783-31--112999 L_4.9.699666... uttst
//doi.org/ 10. 0978 3 319- 71249- 9_46
[27] Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Newrul Networks, 106:249-259, 2018. doi: 10. NEENNE.... 2018,07.011. URL attps:///oll org/ 10. 1016/j neunet 2018 1118.07.. 0011
[28] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Scong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 JERF/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2019, pages 6022-6031. JEEE, 2019. doi: 10.1 10.1109//CC..219.00006 URL htttps::://dd http:::://doooor/1111 org/10
1109/ ICCV 2019. 00612
[29] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effectivv data augmentation with diffusion models. In The Twelfth International Conference on Learning Representations, ICL. 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [RRthttpa:/// hatttpa://ppereviiiiii wwt.:/nee/ooeiiiiecom
ZNzUA9zeAg
[30] Khawar Islam, Muhammad Zaigham Zaheer, Arif Mahmood, and Karthik Nandakumar. Diffusemix: Label- preserving data augmentation with diffusion models. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 26-22, 2024, pages 2761 1 1-27620. IEFF, 2024. doi: 10.1 109/CVPR52733. 2024.02608. URI. httppttts:///o: org/ 10. 1009///0P0/2233..... 2024. .22608
[31] Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladien Koltun. CARLA: an open urban driving simulator. In 1s Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings, volume 78 Proceedings of Machine Learning Research, pages 1-16, PMI.R, 2017. WRR.hhttp//prrceeeiing mlr, www.sereeesviee v78/ 278///oovvtteii117 html
[32] Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang. Wei-Chiu Ma. and Raquel Urtasun. Lidarsim: Realistic lidar simu- lation by leveraging the real world. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA. June 13-19, 2020. pages 11164-11173, Computer Vision Foundation / IEEE, 2020. doi: 10.1 10.1109902222 2020.0011...00
URL.
htcpa://openaccess. thecvf...... com wwt./oo..ccceeeeee cttp/ettttttccceeeeeeeeeeeee cetpeettt_v__ee___ee__e Btt___CVRRRRORttt..1.mm htwp://///aaaaaaaaaaaaaa.. LiDARsim Realistic Siw.LDRRRSmmlaaioonnmmmmoo by Leveraging the Real_World
2020 paper html
[33] Vlas Zyrianov, Xiyue Zhu, and Shenlong Wang. Learning to generate realistic lidar point clouds. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII, volume 13683 of Lecture Notes in Computer Science, pages 17-35. Springer, 2022. doi: 10. 10.10077978-3-031-222 -2000000-222-222222
[34] Dacun Lee and Jinkyu Kim. Resolving class imbalance for based object detector by dynamic weight average and contextual ground truth sampling. In IEEE/CVF Winter Conference Applications of Computer Vision, WACV 2023, Waikoloa, HI, USA, January 2-7, 2023, pages 682 691. JEEE, 2023, doi: org/10 100. 1109//ACV566888 202..
10.1109/WACV56633 2023.00075. wttttttts:////ett..
[35] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. CoRR, hbss/1908.09422 2019. URL (ttL:ptt///ariiv.ooo orgg
abs/ 1908/990..000 09492
[36] Aoran
Xiao, Jaaxing Huang. Dayan Guan, Kaiwen Cui, Shijian Lu, and Ling Shao. Polarmix: A general data augmentation technique lidar point clouds. In NeurlPS. 2022.
URL http::/ wwwp.cc/pppaaeaaeee cwtppppr_pppeeeeeeeeeeeeeee httppppee.lepppppee///aa/aaaaeee 2202//hah// 375588177/40011e4497777777777113 wwwe:/e9ee-bsatttteeeeeeeaaaaeee Conference erence. html
[37] Kazuto Nakashima and Ryo Kurazume. Lidar data synthesis with denoising diffuusion probahilistic models. CoRR, abs/2309.00 00256,, 2023. doi: 10.48550/ARXIV.2239900.
arXiv. .09256
[38] Jin Fang, Dingfu Zhou, Feilong Yan, Tongtong Zhao, Feihu Zhang. Yu Ma, Liang Wang, and Ruigang Yang. Augmented lidar simulator for autonomous driving. JEEE Robotics Autom. Lett. 5(2):1931-199) 2020. doi: 2T..0//L....2222999292 UtL/attss://doo org/ 10/g///.... 1109/LRA, 2020. 2969927
13

---

[39] Jeong Joon Park, Peter R. Florence, Julian Straub. Richard A. Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pages 165-174, Computer Vision Foundation / IEEE, 2019. doi: 10.1 10.1109/CVPR.20 9.00025 httpt //openaccess. teecceesss.tevi.eeeoe
content, S_tps________________ 20019/html//ark DeepSDF Learning. Continuous [rmnedDDistance, Sraaed_DDistarre__uttttooooo... wttpe_ooo_hhh_pp_eeeeeeeee Shaape_kepreeentatiin. CVPR_ 2019 _a111_ppper_._ html
[40] Shunsuke Saito, 7eng Huang. Ryota Natsume, Shigeo Morishima, Hao Li, and Angjon Kanazawa.
Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November https::::/ /doii org/ 10
2. 2019. pages 2314-2314, IEEE, 2019. doi: 10.1 111101//...229.0000220 1109/ICCV 2019.00239
[41] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III, volume 12348 of Lecture Notes in Computer Science, pages 523-540. Springer, 2020. 978 3- 0300-5580--00 8_33
doi: 10.1007/978--340 1--00-555880 31. URL /dooi. .rg...or///// /100 1007 /
[42] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 5835-5844, IEEE, 2021. doi: (L109//CCCV49922222000.000 httpa://doi. org/ 10. 1109/ICCV49922
URL
[43] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5460 5469. IEEE, 2022. doi: 10. N0.1109/CVP55268822200 201.6888.22.0003399
20g/10. 10..100 1109/ CCtR///2R.22.... 2022. 2022.00539
[44] Yoonwoo Jeong, Seungjoo Shin, Junha Lee, Christopher B. Choy, Anima Anandku- mar, Minsu Cho, and Jaesik Park. Perfception: Perception using radiance fields. In NeurlPS. 2022.
URL http://papers. wtwp filapeerfilleeppppe/////2 hashh/
Abstract-Dataseta. and Ben.heennchhh..eeeee .ttml html
[45] Johannes L. Schönberger and Jan- Michael Frahm. Structure wwatrre-froommee mrotimmtooon revisited. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30. 2016, pages 4104-4113. TEEE Computer Society, 2016. //àoi.org//1 1109/CVPR.. 2016. 445.
doi: 10. 10.11009CVP...011.444 URL. https
[46] Kai Zhang, Gernot Riegler. Noah Snavely, and Vladlen Koltun. Nerf++; Analyzing and improving neural radiance fields. CoRR, abs/2010.07492, 2020. URL
07492
[47] Yangming Cheng, Liulei Li. Yuanyou Xu, Xiaodi Li, Zongxin Yang. Wenguan Wang, and Yi Yang. Segment and track anything. CoRR, abs/2305.000 006558, 2023. doi: 10.48550/ARNTVV2235005555 URL. https://doi. org/ 10. 48550/ arr/v///rxiv... 065588
[48] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. CoRR, abs/2305./2266666 2023. doi: 10.48550/ARXIV.2305. 33.44.03 URL lttpp:://ooi......oo org/ /000 48550/arXivv 2305. 02463
[49] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- I-to-3: Zero-shot one image to 3d object. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 9264-9275. IEEE, 2023. doi: 10.1 19911CC550000 2023.00853. https:tts:://et //doi. org/ 10. 1109/ 00000//CCV1100... 2023.022220000222 00853
[50] Jun-Ya Yan Zhu, Taesung Park. Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2242-2251. IEEE Computer Society, 2017. doi: 10. 10.1109/CCV.2211.2222 utt.ttps:://aat //dotps://ddi.... org/ 10. 1111/199CC 2017..24
[51] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++:: Deep hierarchical feature learning on point sets metric space. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett, editors, Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5099 5108, 2017. URL https:///cc/seeediiiiie wtw.eeeedinng.nnuiiisss.. http:/aahhhhh/ http://///////000000000002000000 wtw.://ateeeeAtttttttttta..
14

---

[52] Chen Zhao, Jiaqi Yang, Xin Xiong, Angfan Zhu, Zhiguo Cao, and Xin Li. Rotation invariant point cloud analysis: Where local geometry meets global topology. Pattern Recognit., 127:108626, 2022. doi: 10.10160..ATTOG....
WWW.GG/AAACOG..222.0866528 108666. URL https://d/// otg/ patco.............. 2022. 108626
[53] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjic Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointmet++ with improved training and scaling strategies. In NewrIPS, 2022. URL http:://ppppr:
Saaper__fileeeee wwwp:/tappeeeee/eeeeepepppppeeet httppppprrr/////a///ppp h0s2/hash// Abstractt-trte.ee..e Conf www.oeeeeeeeeiiiieoeoooeeeeeeeee html
[54] OpenPCDet Develoopmen Team, Openpedet: An open-source toolbox for 3d object detection from point
clouds. httpp://ggithu....
com/ httpaaw..oo//ppaaaaaaeaee httpt//ww..cooopppmmmaaaaaaaaaaa httppp////gttt...cc.co.oooomabpa http://nnnmmmaaappppeeeeeeeeee 2020.
[55] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang. Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 1080- 1089, IEEE, 2022. doi: AAL100/CVVR52228.22222200 00116
URL https:
//doi /rdoi.or//// org 100.111////PR26B8
[56] Fabian Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. Se(3)- : 3d roto- translation equivariant attention networks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, De- cember 6-12. 2020, virtual, 2020. URL Awtes4-bbbtttttttteeeeee
nwwrips cc/paper 2020// hash/
[57] Yin Cui, Menglin Jia. Tsung-Yi Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based effective number of samples. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 9268- URL
9277. Computer Vision Foundation IEBE, 2019. doi: 10.1109/CVPR. 2019.00999 http://openaccess. thecvf. com wowceetttttnttceeoeeeeeee 2019/html/Cu /Capa/CCllaaassaaaaeeaaaaaaaaaea Class
Class-Balanced Balanced_ Loss_
Nuw.eeree_uueeeeeeeee_ee Samples,__ CV____CPR__________ CVP._2019_________e2_2222
wat__pppperrtt. html
[58] Hyungtae Lim, Minho Oh, and Hyun Myung. Patchwork: Concentric zone-based region-wise ground segmentation with ground likelihood estimation using a 3d lidar sensor, IEEE Robotics Autom. Lett., 6 (4):6458-6465, 2021. doi: 10.1
httppa//a///o: org//0....g/1... 1109/LRA. 2021
3093009
[59] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL httt:://rxiv, org/ abs/ abs/1412 1412
6980
[60] Xudong Mao, Qing Li, Haoran Xie, Raymond Y. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In IEFF International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2813-2821. IEFE Computer Society, 2017. doi: 10.1109/ ICCV:2017.304. Uttttpps://doo. org/ 10200.1..11110 1ww9/ICCV..... 2017.3004
[61] Seungjae Lee. Hyungtae Lim. and Hyun Myung. Patchwork++:: Fast and robust ground segmentation solving partial under--sementttion using 3d point cloud. In JEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2022, Kyoto, Japan, October 23-27, 2022, pages 13276-13283, IEEE, 2022, doi: D0.100//R0047612.2200 2022. 9981561. URL httppatt:://doo 00g/10. 1109/1R0547612 2022. 9981151
[62] Alexander Lehner, Stefano Gasperini, Alvaro Marcos- Marcos-Ramiro, Michael Schmidt, Mohammad Ali Nikouci Mahani, Nassir Navab, Benjamin Busam, and Federico Tombari. 3d-vficld: Adversarial augmentation of point clouds for domain generalization in 3d object detection. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 17274-17283, IEEE, 2022, doi: 10.1109/CVPRS2668 2122.0118.
URL CtR.tttpt:///// http::://...rrg//1111100000 111p11..111/////2266660
[63] Xuzhong Hu, Zaipeng Duan, Xiao Huang. Ziwen Xu, Delie Ming, and Jie Ma. Context aware data augmentation for LIDAR 3d object detection. In IEEE International Conference on Image Processing. ICIP 2023, Kuala Lumpur, Malaysia. October 8-11, 2023, pages 11-15. IEEE, 2023. doi: 10. 1099 ICIP49359.209 002.10022777. BtLLhhtps:///oo org/ 1109//CII499658 2023. 10222773
15

---

[64] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotný. Common objects in 3d: Large-seale learning and evaluation of real-life 3d category reconstruction. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 10881-1089) IEEE, 2021. doi: 10.1109/CCC4489222221000111
URL
https: org/ro////oo 10. 1ww.11119II1VEA I1w.9892V292222. 2021.01072
[65] Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling. Found. Trends Mach. Learn., 1 11(1):1-96, 2018. doi: 10. 10.1561/2200000000 URL https
1551//000.6111/00000007
[66] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii and Jéröme Revaud. Dustär: Geometric
3d vision made easy. CoRR, abs/2312. 2023. doi: 1048550ARXXX22322141122 URL https
//doi. 10. 485.44850////// arr:0//rrii... 2312. 14132
[67] Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Danicle Sirigatti, Vladislav Rosov, Angela Dai, Matthias Nicßner. Meshgpt: Generating triangle meshes with decodcr-only transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 19615-19625. IEEE, 2024. doi: 10.109//CVRR52733.2220000 201400085. URL
https: //doi org/ 10. 1109/ 21199CVPR52733 2024. 01855
[68]
Peihao Wang, Dejia Xu, Zhiwen Fan, Dilin Wang, Sreyas Mohan, Forrest N. landola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang. and Vikas Chandra. Taming mode collapse score distillation for text-to-3d generation. In IEEE/CVF Conference Computer Vision and Puttern Recognition, CVPR 2024, Seattle, WA, USA. June 16-22, 2024, pages 9037-9047. IEEE, 2024, doi: 10.1 PR..5053333 2024. 00863. //dopp::///d..../ org/ 10. 1109/CVPR52733 2024
16

---

A Appendix
A.1 Dataset Details
Distribution
Motivation. We provide a distribution of an- notated instances across different classes for three benchmarks (nuScenes [2]. KITTI (18).
km
Pseudo GT
of intancos
10c
and Lyft (17) in Fig. Even in few categories, severe class imbalance is observed in the datasets.
This class imbalance can induce a significant per- formance gap between detecting the majority and minority classes. The original skewed distribu- tion of ground truth data (Blue) is mitigated by adding pseudo- Louuo-LiDAR samples generated by the PGT-Auu pipeline for instances of the minority class (Green).
102
Pedestrien
Orcltt
Majarity
Minority
nuScenes instance Distribution
nuscenes
lytt
Pseudo GT samples
samples
10*
of instarcos
103-
of instances
2.0
Or
Pedeerrian
Barner
Trallen
Bus
Matoroycl
Bloyte
Car
Pedestrion
Blcycle
Dus
Motoryyee
Anima
o.v
Majority
Minority
Majority
Minority
Figure 7: Instance Distribution on various benchmarks.
Data Collections. We first provide the number of videos collected from miniatures and web pages as shown in TO We purchased various miniatures within a budget of around $100, and crawled data using the following keywords Google: 360 camera, surround view, turntable, sales, and second-hand. Finally, the video collection consists of 64 trucks 21 motorcycles" and 10 buse from YouTube, along with additional footage of construction vehicles from the Ritchi Bros Auction website
We provide examples of our dataset collection in Fig.
Table 10: Statistics for our data collection. (i) Videos capturing surround view of miniatures and (ii) publicly available videos of given minor-class objects. Abbr. C.V: Construction Vehicle, M.C: Motorcycle, B.C: Bicycle
Data Source
C.V
Trailer
Truck
Bus
M.C
B.C
Total
Miniatures
10
4
5
3
3
31
Public Videos
2
64
"htps://www.youtet cotet/ccccccc//ooee/eeeeeee MHDSuperSttor
hhtps://www.youttee com/ kagamotors
http:://www.bbautttoo.coom
17

---

Figure 8: Dataset Collection. We demonstrate collection of miniature images and crawled web videos.
18

---

Volumetric 3D Instance Collection. To aid in understanding the operation of the proposed frame- work. we present all results from input images output results including foreground segmentation masks, extracted foreground objects, camera poses, and the corresponding dense RGB-coloree point clouds in Fig.
Web Images
Miniatures
(a) Camera Images
(b) Masks
(c) Foreground
(d) Camera Poses
(e) RGB Point Clouds
Figure 9: Volumetric 3D rendering Results of RGB Point Clouds.
19

---

Object-level Domain Alignment. We demonstrate the effect of our rearranged range projection as shown Fig. T0 Not only qualitative differences, but also by reflecting the sensor configuration, can reduce the discrepancy to the actual data domain.
(a) Vanilla Range Projection Rearranged Range Projection
Figure 10: Examples of Generated Point Clouds (a) with and (b) without rearranged range projection. We use Azimuth resolution: 1080, Vertical FOV: -30" 10 Channels: 32 for sensor configuration.
A.2 Implementation Details
View-agnostic Color Representation. We provide pseudo-code for ad-hoc module obtain representative colors for each voxel grid or point based on the estimated mean color values from different views. We mainly adopt PeRFception 44 framework implement view-agnostic 2D-to-3ID rendering.
Algorithm 1 Pseudocode for estimating mean RGB value of point cloud a pretrained voxel grid or point cloud.
Input (H x W) rays from N views, Spherical Harmonic (SH) coefficients sh, step size tstep-
Spatial indices of the 3D data structure (Voxels or 3D Gaussians)
Output RGB PointCloud
1: for each i-th view in N do
2:
for each j-th ray in H x W do
3:
Define RG count to store RGB value and counts at i-th view. Calculate ray origin and direction rd, To ray rij
4:
5:
Calculate SH basis shtasis with ra/ra-
6:
Calculate the ray Tag" is far bound kmax> and the near bound Lyiin-
7:
Set t as tmin-
8:
while < tmaz do
9:
Extract the position in the range [t, t tstep) of ray Téj.
Obtain spatial index lidz of the position.
11:
Obtain SH coefficients sh. +- sh[[sar]-
Calculate RGB of voxels Gaussians by E(she X shbasis)
13:
Add RGB value and one to RGB sum. count at index l respectively. Increment by step end while
15:
end for
17:
Append RGB sum, and counts of voxels or Gaussians in i-th view to arrays end for
18:
20

---

Front-back Classification. For the front-and and-back binary classifier, we build conventional PointNet- based model and train the model during 100 epochs with binary cross-entropy loss, using Adam optimizer [59] with beta values (0.9, 0.999). The learning rate decreases from 0.001 through the step decay scheduler with a step size of 20 and a gamma of 0.5. We used NuScenes [9] point clouds as the training set and flipped the point cloud with a probability of 0.5 to the model input. The model transfers input data into 128-dimensional feature vectors. These are then mapped into scalar probability to classify whether the vehicle is heading forward or backward.
LiDAR Intensity Simulator. Our model is based on CycleGAN 50 for unpaired translation between RGB and intensity, including class- PointNeXt [53 generators and discriminators. For the intensity domain, we randomly selected 300 target class samples from nuScenes dataset [9]. We then generated another 300 samples for the RGB domain by performing dee-dependeent point filtering on RGB dense point clouds from 2D-to-3D renderers according to the location and heading of the nuScenes sample. For batch training, convert input samples more than 256 points into samples with 256 points using farthest point sampling. Consequently, we trained the intensity simulator for each target class over 200 epochs with batch size 16, using Adam optimizer (0.5, 0.999) and learning rate of 0.001.
[59] with beta values
For two generators D'DnssDDun: GDin:-Drghs and two discriminators DD. Dr weighted loss consisting of group intensity loss, cyclic loss, and LSGAN loss [60].
L (GDru GDia Drrri DDine) =
LISGAN (GD..g C.D.igb-Den -Dni Drma Dint; Drgb)
+LLSGAN Dim Dr Drgb: int) +Leye (GD G + -oais-DDrgg) +XGgroup (GD D int
(4)
D Drgb))
A.3 Experiments Details
Pseudo Object Bank Details. To verify the impact of the proposed pipeline on the 3D object detection task nuScenes dataset (9) created a pseudo object bank from RGB points generated by Plenoxel [14. We used the intensity estimator trained from the point cloud with RGB features, with group intensity loss and CycleGAN loss, and performed point filtering and rearrangement according to the sensor configuration of the nuScenes dataset [9]. This pseudo object bank has the
FID score 7.7 on the nuScenes GT object bank (see Table 4. For each dense RGB point cloud object, our 2D-to-3D renderer, including view-agnostic color extraction, took approximately 20 minutes. Training the intensity simulator for each class took approximately 30 minutes, and the generation of pseudo object bank samples, which includes axis aligmment and projection, took I hour. These processes were completed using AMD EPYC 7313 16-Core Processors and an NVIDIA A100.
Detection Models. We provide all the YAMI. configuration files for the OpenPCDet (54) models used the paper. Note that we only modified the augmentation part of the configuration for fair comparison. We adopt Adam optimizer with one-cycle learning rate policy and Patchwork++ [61] to obtain ground points of the input scenes. Note that SECOND ( was performed on a voxel with a size of [0.1, 0.1, 0.2). The CP-Pillar [5] was performed on voxel with a size of [0.1, 0.1, 8). while the other remaining models were performed with a voxel size of [0.075, 0.075, 0.2]. Training each model took approximately 36 hours using four Nvidia A100s, 256Gi memory and 4 AMD EPYC 7313 16-Core Processors on a single workstation.
A.4 Ablation Studies Details
To demonstrate the effect of the quality and quantity the pseudo object banks, we performed various ablation studies. All experiments for the 3D object detection task are conducted based on Co-Voxel [5] operating on the input voxel size [0.075, 0.075, 0.2], while maintaining the setting of the remaining hyperparameters from OpenPCDet [54].
21

---

Quality of Pseudo Labels. We conducted experiments to assess how plausible and realistic our generated object points are compared to existing real-world datasets. To measure FID scores, we use transsorrre [56] model trained on nuScenes dataset over 100 epochs with batch size 4, using Adam optimizer [59] with a learning rate of 0.002 and a weight decay parameter of 0.5. We first sort the objects in nuScenes in descending order based on the number of points, divide them into 32 groups, and sequentially sample objects with many points, resulting in class sets with 593 objects. Note that A2D2 [18], Lyft [17], and our pscudo GT bank use all available samples. The class mapping between different detection datasets is shown in Table TT In feature extraction, the object is sampled with 64 points, followed by local grouping with the nearest 32 points. These features are inputted into the trans-tranfoommer [56] to extract the object- aware feature. Our implementation codes originate from httpt:ttttt:tttiieeeee www.co/////////
Table 11: Categories agreement among datasets for FID evaluation and 3D object detection.
A2D2 Dataset [1]]
Lyft Dataset 17
nuScenes Dataset
Truck
Truck
Truck
Bus
Bus
Bus
MotoBiker, Motorcycle
Motorcycle
Motorcycle
Bicycle. Cyclist
Bicycle
Bicycle
Trailer
Trailer
Utility Vehicle
Construction Vehicle
Animal
Caravan Transporter
Emergency Vehicle
Other Vehicle
Samples from Other Datasets. Besides pseudo LiDAR generation for nuScenes dataset, we also generated object banks for KITTI [18) and Lyft [17 to demonstrate PGT-Aug's effectiveness on
different detecction benchmarks. We adopt perception range of x from -40 to 40m and y from to 70m at the interval of 5m for KITTI [18), and adopt range from -80m to 80m for Lyft [17] according to dataset specific evaluation schemes. We use 13 heading directions (from -880 to 180 at intervals of 30"), We augment pseudo LiDAR for [cyclist] class for KITTI [18,, and augment [truck, motorcycle, bicycle, bus, other vehicle] classes for Lyft (17).
A.5 Efficiency and Real-time Applicability of Pre-trained Models
Memory usage and inference time If Object-level Domain Alignment is performed during detection training. it may introduce additional time overhead compared to GT--uug or Real-Aug. In 3D object detection methods, the objects to be inserted are generated and stored prior to training, and these stored objects are loaded and inserted during the training phase. The process of loading these objects from disk into memory for scene insertion incurs time, and the memory and time complexity per batch is ((n) for the number of objects n to be inserted, similar to Real-Aug. As result, real-time performance was not a primary consideration in our approach.
We would like to emphasize two key points: (1) the pseudo LiDAR point clouds are generated and stored offline, and (2) these generated point clouds are loaded from memory and augmented during the training of 3D object detectors. Therefore, the requirement to run pre-trained models (e.g., unpaired domain transfer models) in real-time becomes less critical. In other words, we would like to emphasize two key points: (1) the pseudo LiDAR point clouds can be stored offline, and (2) the insertion of sampled objects from ground truth instance bank is already common practice, in typical 3D object detector training. Additionally, in Table [2] we analyze the processing time and memory usage for generating pseudo LiDAR point clouds across various object classes, such as construction vehicles, trucks, and trailers. This analysis confirms that our pipeline can efficiently generate point clouds, with a total processing time of less than 300ms.
22

---

Table 12: Average processing time (per instance, in msec) and memory usage (MB) C.V. Con- struction Vehicle, Ped: Pedestrian, T.C: Traffic Cone, M.C. Motorcycle, B.C. Bicycle.
Minority classes
Bus
C.V
Trailer
Truck
M.C
B.C
Intensity Estimation
150
140
250
40
34
178
View Dependent Point Sampling
67
44
40
30
6
78
Rigid body motion
8.80
8.65
8.6
8.53
8.55
9.09
Memory
4.006
4.052
4.098
4.013
4.012
4.074
A.6
Qualitative Analysis for 3D Detection Task
We provide additional qualitative results for the model trained with the proposed PGT-Aug in Fig.
18)
Brukk Traker Diydl
BLR
(8)
(
(8) GT Box
Real-Aug
(e) PGT-Aug Pred Box
Figure 11: Detection output comparison between Real-Aug and PGT-Aug (ours). We provide bounding boxes of (a) ground truth and detected by (b) Real-Aug 12 and (c) PGT-Aug.
23

---

B Additional experiments.
B.1 Additional Ablation Studies.
Comparison with using only first spherical harmonic coefficient. As shown in Table 13 compare our method with a variant model (where Plenoxel's spherical harmonic (SH) coeffcients are set to 0) to see the quality of generated pseudo LiDAR point clouds in terms of FID score. The experimental results confirm that our approach can generate point clouds that are more similar the real LiDAR data.
we
Table 13: FID evaluation between SH coefficient 0 and ours. Abbr, C.V: Construction
Vehicle, Ped: Pedestrian, T.C: Traffic Cone, M.C. Motorcycle, B.C: Bicycle.
Bus
C.V.
Trailer
Truck
M.C
B.C
Average FID
SH coefficient 0
14.9
13,2
8.0
7.2
2.3
3,0
8.1
Ours
13.2
13.2
7.6
7.3
2.1
2.1
7.7
Ablation study on the impact of pre-trained models. To assess the impact of pre-trained models, we conducted additional experiments under the following conditions: (A) constant intensity without intensity generation, (B) random sampling without view-dependent point filtering, (C) rigid motion model, and (D) and (E) instance generation based on changes in the number of images. As shown in Table [4] the removal of intensity generation in pseudo LiDAR led to performance decline in downstream detection tasks. Moreover, as the percentage of multiview images used for 3D reconstruction decreased, the overall detection performance decreased accordingly. Lastly, we performed experiments on general alignment by replacing object-level alignment with random sampling of points from dense RGB point clouds and removing the motion models. Both experiments demonstrated suboptimal performance compared to PGT-Aug.
Table 14: Ablation study in intensity, the number of images, data alignment on nuScenes val set.
Abbr. C.V: Construction Vehicle, Ped: Pedestrian, T.C: Traffic Cone, M.C. Motorcycle, B.C: Bicycle. t: our reproduction.
Majority classes
Minority classes
Experiments
Ped
Barrier
Trailer
Truck
M.C
mAP
NDS
Constant Intensity (A)
85.4
85.0
68.1
71.5
72.6
23.6
38.9
60.0
68.3
54.2
62.75
68.75
Random sampling (B)
85.4
85.1
67.2
71.4
73.2
22.8
39.2
59.8
68.1
55.4
62.76
68.56
No-motion (C)
85.4
85.2
67.8
71.5
60.1
69.0
56.6
63.22
68.80
25% the number of image (D)
85.3
85.0
67.3
71.1
73.4
23.1
40.1
59.2
70.2
56.4
63.12
68.78
50% the number of image (E)
85.5
84.9
68.5
71.4
72.0
23.1
41.4
59.8
69.5
56.4
63.26
69.03
PGT-Aug
85.4
85.4
68.0
71.1
72.1
24.2
40.4
59.8
70.3
58.3
63.52
69.11
B.2 Effect of PGT-Aug the majority of classes
Our primary objective is to generate and insert the minority classes instead of the majority classes relieve the class imbalance issue. Thus, we anticipated that the performance of the minority would improve, while the performance of the majority classes would either remain stable or improve slightly due the synergy effect from addressing the imbalance. To verify the effectiveness of the pipeline to majority classes, we compare PGT-Aug with other 3D data augmentation methods, including LiDAR-Aug [10), 3D-VField [62]. and CA-Aug (63), the KITTI (18) Car benchmark. To match their baseline, we conducted the experiments on PointPillars 3 We reconstructed 10 cars of 3D RGB point clouds from CO3D dataset (64] and created about 16,000 pseudo LiDARs of Car class. We apply pseudo LiDARs along with GT LiDARs during augmentation, and our PGT- Aug performance largely outperforms GT-Aug, Real-Aug, and other augmentation methods in the car
detection benchmark shown in Table 15
Instance sample details. As shown Fig. 12 we create pseudo LiDAR point clouds for method by randomly sampling ten from the CO3D dataset. The instances used

---

Table 15: Performance comparison with other data augmentation approaches. 1 our reproduc-
tion
AP car 70 (40% recall)
Method
Easy
Moderate
Hard
Baseline
88.08
74.85
70.55
GT-Aug
87.80
78.36
75.41
LiDAR-Aug 10
87.75
78.24
75.35
Real-Aug
12
88.13
78.97
76,06
3D-VField
87.05
77.13
75.55
CA-Aug
63
88.82
78.66
75.75
PGT-Aug
90.00
79.45
76.35
as follows: 106 12650 23736, 106 (266122622226 23043, 157 1171_172662211 33548,
17286_33548, 185 19982_37617
194 14__20899__0 41094, 216 227962226662 -7444, 226_22227___2 __442,442222 421 (00___84055111211
and 421 421_58407 112553.
Figure 12: Examples of pseudo GT samples generated by our pipeline for the Car class from the CO3D dataset.
B.3 Discussion Towards Better Performance
We introduce some novel techniques for improving the effectiveness of PGT-Aug that were not applied in the main paper for fair comparison in Table We expect that these methods will be helpful for future research. these experiments, the same pseudo-object bank in Section 4 was used.
Ray tracing and Distance filtering. Ray tracing can simulate partially occluded form of objects when obstacles exist between the newly inserted object and the location of the LiDAR sensor by a range projection. Through this method, observed that ground truths located outside the perception range can increase the class uncertainty due to sparsity. Therefore, ground truth bank. In this experiment, we set a distance threshold of 54m.
generate realistic scenes. For distance filtering,
reduce the sample uncertainty by eliminating those samples in the
Bandit Algorithm for Object Placement. The advantage of our pipeline is that it can generate fully volumetric objects, allowing it to be inserted at any location. To maximize the advantage, we used Thompson sampling (65] determine the candidate positions for pscudo LiDAR samples during training. We formulate this position decision as bandit problem of selecting cells of grid representing a quantized space around the ego vehicle. First, for cells with the ground truth box, the confidence score of the prediction is greater than 0.5, the detector is considered to have performed well in the cell area and the success count of the cell increases. On the other hand, if the confidence score of the negative samples is greater than 0.5, it is considered a failure, and the failure count of the cell increases. For each cell, Thompson Sampling is performed by setting the alpha of the beta distribution as the failure count and the beta as the success count. After that, the top cells with
25

---

the highest probability are selected. This process allows us to insert objects where the detector fails during training. We set the grid size 0.075 8m in this experiment.
Table 16: Detection performance comparison for additional modules on nuScenes val set in
terms of AP, mAP, and NDS. We use CP- Voxel S and Transfusion-L. [55
baseline model.
CP- Voxel [5
Transfusion-L [55
Model
A
B
E
F
G
H
w/ PGT-Aug
/
/
/
/
/
/
v
Ray Tracing
/
/
/
/
w/ Distance Filtering
X
X
/
X
X
/
/
W/ Thompson Sampling
X
X
X
X
X
X
/
mAP (1. for all 10 classes)
63.5
63.8
63.6
63.7
64.2
65.0
64.4
65.2
NDS for all 10 classes)
69.1
69.3
69.0
69.3
68.8
69.2
68.7
69.4
C Significance of the Proposed Method
In 3D object detection, there have been studies aimed at addressing class imbalance problems by modifying model structures (CBGS) [35] or adjusting the loss function [34]. The most widely used method is the over-sampling known GTT-Aug, which involves inserting objects from other frames into the current frame. However, this method is limited to inserting objects from predefined pool (training set), which restricts learning intra-class diversity, and generating collecting 3D data for various objects has been extremely challenging and expensive. Recent advancements in differentiable 3D reconstruction techniques, such as NeRF and Gaussian Splatting, have made possible to reconstruct dense 3D points at a lower cost. By leveraging these 3D reconstruction techniques, we proposed novel and practical pipeline that overcomes the limitations of traditional insertion methods, particularly in terms of cost and diversity. Also, our pipeline is not limited to specific generative model and can be applied to any renderers.
C.1 Generalization by Different Renderers.
To prove the capability
proposed framework, we show the generated RGB-colored point clouds and the simulated LiDAR point clouds of the yellow loader by different renderers: Plenoxels [14]. Gaussian-Splatting
Gaussian-Se Splatting [16), and DUSt3R (66) in Fig. I3 Even when different models are applicd, we
observe that our proposed framework can consistently generate high-quality LiDAR objects.
2D-3D Renderers
RGB Point Clouds
nuScenes
KITTI
Lyft
Plenoxels
Gaussian Splatting
DUS:3R
Figure 13: Examples of RGB Point Clouds and Generated Pseudo LiDAR from Various 2D to 3D Renderers.
26

---

C.2 Reason for using Neural Scene Representations, not Generations.
In order place the restored objects in various positions, necessary restore the entire shape of the object and create bounding box. We chose explicit 3D representation models (3DGS Plenoxel, or various NeRF methods) that can reliably reconstruct the entire shape and perform robustly with relatively small objects, rather than Text-to-3D Generation models (Shap-F 48), MeshGPT [67]. etc.), which tend to collapse modes when the target image distribution overly concentrated in single peak, resulting in abnormal 3D objects that are strongly related to specific views (Janus problem) 688
C.3 Data Size.
Our dataset is created to provide pseudo-LiDAR point clouds of minority-clase objects, which can be augmented into typical driving datasets compensate for the class imbalance problem. Thus, the volume of our dataset should be smaller than that of these datasets but sufficient to compensate for the class imbalance problem, as we reported in our experiments. Moreover, our approach offers significant advantages in data generation and flexibility: (i) We provide an automated pipeline to generate pseudo-LiDAR point clouds, enabling the community to easily produce and expand datasets with various objects. (ii) Our method can generate view-dependent Eseudo-LiDAR point clouds that can flexibly placed anywhere in the scene. These features allow for rapid and efficient dataset expansion, significantly improving data efficiency and enabling researchers to quickly increase their dataset size with minimal effort.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims in abstract and introduction clearly elaborate on our work's moti- vation to solve class- imaalance in 3D LiDAR-bases detection and three contributions. We also provided experimental results certify our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We addressed the limitations our synthetic datasets and pipeline in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
28

---

Answer: [NA].
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide a detailed explanation of the methodology for reproducing our results and experimental settings and demonstrate the robustness of our approach through various experimental results. Through fixed seed, we ensure deterministic behavior in all experiments. More details on the implementation are provided in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
29

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We attached dense RGB point cloud data, part of the pseudo-LiiDA object bank, and a visualization code as the supplemental material. Our code and data will be made available through the project page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ctipss.c//ublic//uuie guides/ CodeeSumiiiseeeeee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified our details of training and test datasets, and details of experimental settings such batch size, hyperparameters, hardwares, codebase, size of synthetic datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We answered no but created the fixed object bank for augmentation eliminate randomness. Through a fixed seed, we ensure deterministic behavior in all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We clearly address the computer resources for our experiments in both main paper and supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:s////orrrss.ccc/ubll/tttlta
Answer: [Yes]
Justification:
We have reviewed our work according to NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have addressed societal impacts of our work in the paper.
31

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our released datasets and detector checkpoints do not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We specified the source of scraped real-world videos of long-tail classes and properly cited the use of existing models, code package, and dataset. The source code of PeRFception and OpenPCDet have Apache License 2.0 and the source code of CycleGAN has BSD License, and Segment-and Bogment-and-Track- Trach-An has AGPL-3.0 License. The corresponding license is properly inserted in each readme.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
32

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided along with the assets?
Answer: [Yes]
Justification: We provide detailed description of our newly created pseudo-LiDAR object bank in the main paper, alongside analysis on integration of new assets with existing detection pipeline. We also provide README instructions that include execution methods for the source code and folder structure for our object bank.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
33

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
34


Reviews:
[{"review_id": "PT4jHlOWkH", "reviewer": "Reviewer_iAKq", "summary": "This paper introduces Pseudo Ground Truth augmentation (PGT-Aug) to address class imbalance in LiDAR-based 3D object detection. PGT-Aug generates diverse pseudo LiDAR point clouds from low-cost miniatures or real-world videos and involves three steps: 3D instance reconstruction, object-level domain alignment, and context-aware placement. Extensive experiments on nuScenes, KITTI, and Lyft benchmarks demonstrate its effectiveness, especially for datasets with large domain gaps.", "questions": "How was the map information (L200) obtained?", "limitations": "no potential negative societal impact of their work", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Originality & Practical Impact**: Introduces PGT-Aug, a novel, cost-effective pipeline for addressing class imbalance in LiDAR-based object detectors using pseudo-LiDAR samples from videos and miniatures.\n- **Quality & Clarity**: Offers a thorough methodology with clear explanations and robust evaluations, enhancing the accuracy and robustness of object detectors.", "weaknesses": "- As seen in line 532, this paper does not collect as much data compared to large datasets like KITTI. I question the benefits and improvements brought by this work in terms of data collection.\n- The proposed method requires the use of many pre-trained models, such as the unpaired domain transfer model (L179) and a rigid body motion model (L217). First, the computational efficiency and real-time applicability of these added models need to be addressed. Given the added complexity of the aforementioned models, understanding the computational trade-offs and optimizations required for practical deployment is essential (e.g., the memory usage and inference time), but these aspects are not sufficiently covered in the paper. Second, the impact of the performance of these pre-trained models on the proposed method needs to be discussed and evaluated.\n- The authors use YouTube videos and \"crawled data using the following keywords on Google\" (L533). They should obtain permission from the data/content owners. Simply citing the sources in the paper is clearly not sufficient.\n\nMinor issue: \n- \"and r is 0.1\" should be better move to L.190, since there are no r in Eq. (3)\n\nOverall, I really like the interesting idea in this paper. If the authors can address my issues in the rebuttal, I am willing to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n---\n**[W1] Data Size**\nOur dataset is created to provide pseudo-LiDAR point clouds of minority-class objects, which can be augmented into typical driving datasets (e.g., nuScenes, KITTI, and Lyft) to compensate for the class imbalance problem. Thus, the volume of our dataset should be smaller than that of these datasets but (we believe) sufficient to compensate for the class imbalance problem, as we reported in our experiments. Moreover, we would emphasize that (i) we provide an automated pipeline to generate such pseudo-LiDAR point clouds, enabling the community to use it to produce continuously growing datasets with various objects. (ii) We can generate view-dependent pseudo-LiDAR point clouds that can flexibly be placed anywhere in the scene, significantly improving data efficiency. We will clarify this in the final version of this paper. \n\n---\n**[W2-1] Efficiency and Real-time Applicability of Pre-trained Models**\nThis is a good question. We would emphasize that (1) pseudo LiDAR point clouds are generated and stored offline, and (2) these generated point clouds are loaded (from memory) and augmented during the training of 3D object detectors. Thus, the need to run pre-trained models (e.g., unpaired domain transfer model) in real-time would be less significant. Further, in the following Table I and II, we analyze the processing time of each model and memory usage for generating pseudo LiDAR point clouds of different classes, i.e., construction vehicles, trucks, trailers, etc. This confirms point clouds can efficiently be generated through our pipeline, taking less than 300ms in total. We will discuss this in detail in the final version of this paper. \n\n**Table I. Average Processing Time (per instance, in msec)**\n| Class | C.V | Truck | Trailer | Motor | Bicycle | Bus |\n| :---- | -----: | -----: | -----: | -----: | -----: | ----: |\n| Intensity Estimation | 150 | 140 | 250 | 40 | 34 | 178 |\n| View Dependent Point Sampling | 67 | 44 | 40 | 30 | 6 | 78 |\n| Rigid body motion | 8.80 | 8.65 | 8.60 | 8.53 | 8.55 | 9.09 |\n\n**Table II. Average Memory Usage (MB)**\n| Class | C.V | Truck | Trailer | Motor | Bicycle | Bus |\n| :---- | -----: | -----: | -----: | -----: | -----: | -----: |\n| Memory usage | 4.006 | 4.052 | 4.098 | 4.013 | 4.012 | 4.074 |\n\n---\n**[W2-2] Ablation Studies with Pre-trained Models** \nTo demonstrate the impact of pre-trained models (e.g., unpaired domain transfer model and rigid body motion model), we further conduct ablation studies as follows: (1) We compare ours with a variant model without luminance generation (i.e., using constant intensity to see the impact of the unpaired domain transfer model). (2) We compare ours with a variant model without our rigid body motion model. In Table III below, we observe both experiments confirm the impacts of using these pre-trained models, showing a degradation without these models. We will discuss this more thoroughly.\n\n**Table III. Ablations in luminance generation and motion model**\n|  | Car  | Ped | Barrier  | T.C.  | Bus  | C.V. | Trailer  | Truck  | Motor  | Bicycle  | mAP | NDS |\n| :---- | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n| Ours | 85.4 | 85.4 | 68.0 | 71.1 | 72.1  | 24.2 | 40.4  | 59.8  | 70.3  | 58.3  | 63.52  | 69.11 |\n| with constant intensity \\[1\\] | 85.4 | 85.0 | 68.1 | 71.5 | 72.6 | 23.6 | 38.9 | 60.0 | 68.3 | 54.2 | 62.75 | 68.75 |\n| without rigid body motion model \\[2\\]| 85.4 | 85.2 | 67.8 | 71.5 | 71.9 | 23.7 | 41.0 | 60.1 | 69.0 | 56.6 | 63.22 | 68.80|\n\n\n\n---\n**[W3] Data/content permission issue**\nThis is a good comment. We also take this copyright issue seriously. First of all, we will not download and re-release those video sources. Instead, we will release a list of YouTube links. Further, as the reviewer suggests, we have contacted the copyright holders to obtain permission to share their video links publicly. We will make sure to resolve this copyright issue upon publication. \n\n---\n**[W4] Minor Comment**\nWe agree with the reviewer. We will move \"and r is 0.1\" to L190.\n\n---\n**[Q1] Map information**\nnuScenes dataset provides the BEV map, annotating commonly-observed map features such as road segments, lanes, crosswalks, walkways, stop lines, and parking lots. For all scenes, we generate an ego-vehicle-centered map in a top-down coordinate system, and over 34k maps are generated. Note that a range of 102.4m x 102.4m is set for generating a map where ego-vehicle is centered (effective forward sensing range of ego-vehicle is 51.2m). KITTI and Lyft datasets do not provide such map information (as mentioned in L291). Thus, we utilize a LiDAR-based ground segmentation method called Patchworks++ [A] to estimate ground. We will further clarify this in the final version of this paper.\n\n[A] Patchwork++: Fast and robust ground segmentation method for 3D LiDAR scans (IROS 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tpuZOlS8sL", "reviewer": "Reviewer_eXGm", "summary": "This paper introduces Pseudo Ground Truth Augmentation (PGT-Aug), a novel data augmentation technique for LiDAR-based 3D object detection. The goal of PGT-Aug is to address class imbalance in training datasets by generating diverse point clouds for minority-class objects.\n\n- **PGT-Aug:** a novel cost-effective pipeline for LiDAR-based object detectors to solve  the class imbalance problem by\n    - (i) generating pseudo-LiDAR samples\n    - (ii) augmenting them during training to balance the performance gap across classes.\n- **Reduce the domain gap:** use spatial distribution matching and data-driven intensity adjustments to achieve\n- **a novel map-aware augmentation technique:** placing an object into the appropriate  locations in the given scene).", "questions": "1. About data collection. For public videos from YouTube can you deal with the dynamic object. \n2. In line 138-140, I know Plenoxels and 3DGS are view-dependent representations, but why it is not fully visible or uniformly high-density. \n3. what is the degree of spherical harmonic coefficients author use. Why not just use to zero degree of SH to solve question 2.\n4. Now there are a lot of autonomous driving simulator work or object-level NeRF/3DGS reconstruction work or shape-gpt/mesh-gpt, can also achieve small objects clone and modify, you can talk about your work and their differences and advantages.\n5. NuScenes tests whether contains information about the added object\n\n**I'm willing to change the grade if address my concerns.**", "limitations": "The author has said it clearly in the article limitation", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Writing is good. A cool manuscript name.\n2. Ample experiment.", "weaknesses": "1. There is code provided, but it lacks documentation and is difficult to use to help understand and visualize the results\n2. Lack of novelty(not sure), see details in question 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n---\n**[W1] Code Documentations**\nOur apologies. We will revise the current documentation and make them easy-to-follow for researchers to use and understand our code easily. Plus, we will add a Jupyter Notebook-based tutorial to guide (i) how to reproduce our model, (ii) how to use our code, and (iii) how to visualize figures (shown in the paper).\n\n---\n**[Q1] Data Collection from Videos with Dynamic Objects**\nThis is a good question. Our method relies on SfM to optimize camera poses for 3D reconstruction. However, such SfM would not triangulate well with images containing dynamic objects. This would be why we reconstruct the 3D shape of stationary objects, followed by using a rigid body motion model to represent dynamic objects. However, this must be an interesting direction worth exploring in our future work.\n\n---\n**[Q2, Q3] View Consistent Representation**\nThis is a great question. We agree that the zero-degree spherical harmonic would have a similar effect to ours. However, we do not use it because (1) we empirically observed that our approach is more robust at capturing the original object's color and areas of dark shades (see our Figure I in rebuttal supp). (2) We wanted our model more generally applicable to various generative models, which may and may not use spherical harmonics.  Additionally, (3) in terms of FID score, we observe our approach is generally better. As shown in Table IV, we compare ours with a variant model (with plenoxel's SH coefficient set to 0) to see the quality of generated pseudo LiDAR point clouds in terms of FID score. Our experiment further confirms our approach generates point clouds more similar to the actual LiDAR points. We will add discussion on this.\n\n**Table IV. FID score evaluation between SH coefficient 0 and ours**\n|  | Truck  | Bus  | C.V. | Trailer  | Motor  | Bicycle  | Average FID |\n| :---- | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n| SH coefficient 0 | 14.9 | 13.2 | 8.0 | 7.2 | 2.3 | 3.0 | 8.1 |\n| Ours | 13.2 | 13.2 | 7.6 | 7.3 | 2.1 | 2.1 | 7.7 |\n\n---\n**[W2, Q4] Differences to previous simulator works or generative models**\nBy leveraging current 3D reconstruction methods, our main goal is creating a novel and practical pipeline that overcomes the limitations of existing 3D object detections such as class imbalance problems, lack of annotations, in terms of cost and diversity. In other words, our pipeline is not limited to a specific generative model and can be applied to any model.\nThe reasons we used explicit 3D representation models (3DGS or Plenoxel) in the paper instead of the models mentioned by the reviewer are as follows. According to [B] (see section Limitations and Failure Cases in supplementary material), Text-to-3D Generation (Shape-GPT, mesh-GPT, etc.) models tend to collapse modes when the target image distribution is overly concentrated in a single peak, resulting in abnormal 3D objects that are strongly related to specific views (Janus problem). Also, as shown in Figure II of Rebuttal-Supp., recent work of multi-view 3D reconstruction pipeline, DUSt3R [C] often fails to recover the details of miniature-scale objects. However, in order to place the restored objects in various positions, it is necessary to restore the entire shape of the object and create a bounding box. Therefore, we chose models that can reliably reconstruct the entire shape and perform robustly with relatively small objects.\n\n[B] Taming Mode Collapse in Score Distillation for Text-to-3D Generation (CVPR 2024)\\\n[C] DUSt3R: Geometric 3D Vision Made Easy (CVPR 2024)\n\n---\n**[Q5] NuScenes Setting**\nAll detectors were tested under the same conditions, meaning no further information about our generated objects was used during the evaluation process, i.e., our model only uses the generated objects during training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xul8Qn2cIK", "reviewer": "Reviewer_kswe", "summary": "The paper presents a novel pipeline for LiDAR-based object detectors to solve the class imbalance problem by generating pseudo-LiDAR samples (from multi-view images of miniatures and public videos of an object) and augmenting them during training to balance the performance gap across classes. The augmentation technique proposed in this paper demonstrate the superiority and generality on nuScenes, KITTI and Lyft datasets.", "questions": "1.\tThere are many works dealing with long-tail problems or class imbalance that are not based on data augmentation. Can authors discuss their application to this problem?\n2.\tCompared to Real-Aug, how is the time cost of the proposed method (PGT-Aug) while achieving less than a 1-point increasement in both metrics mAP and NDS?", "limitations": "The authors discussed the existence of domain discrepancies in both datasets and categories.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe augmentation technique proposed in this paper demonstrate the superiority and generality through performance improvements in extensive experiments conducted on popular benchmarks, i.e., nuScenes, KITTI, and Lyft, especially for the datasets with large domain gaps captured by different LiDAR configurations.\n2.\tThe paper is well-written and easy to follow, especially the part explaining the background. \n3.\tIt presents good experimental results and intuitive visualizations, convincingly demonstrating its effectiveness.", "weaknesses": "1.\tThere is a lack of comparative experiments with other methods that aim for class imbalance problems.\n2.\tThe motivation of this paper is not clear. There is a need to discuss the necessity of using data augmentation methods to solve class imbalance problems. Why can't we use some loss-based or strategy-based methods to handle class imbalance issues?\n3.\tThe detail of the framework is not clear. For instance, there is an Intensity Domain Alignment in the framework, but what it is in detail? e.g., the structure of it, and how it works.\n4.\tIs PGT-Aug not friendly to the majority of classes? As shown in Table 2, PGT-Aug's performance is close to that of Real-Aug for the majority of classes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n---\n**[W1, Q1] Comparison between other methods that aim for class imbalance problems**\nAs the reviewer pointed out, there are other comparative methods dealing with class imbalance using loss-based methods [D, E] without adding data. We attach the comparison results with [D, E] in Table V. To match their baselines, we conducted experiments on PointPillar model. Additionally, we re-implemented Class-Balanced Loss [D] with beta 0.999 and resampled the number of objects. While we find out that loss-balancing methods such as Dynamic Weight Average [E] and CB Loss [D] are effective in enhancing minority-class performance, PGT-Aug, a data-augmentation based method, brings the largest performance gain against other approaches. Also, [E] experimentally proves that its GT sampling-based data augmentation was more effective than loss-based method in improving detection performance. \n\n**Table V. Comparison between other methods that aim for class imbalance problems**\n|  | Car  | Ped | Barrier  | T.C.  | Truck  | Bus  | C.V. | Trailer  | Motor  | Bicycle  | mAP |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| CBLoss  \\[D\\] | 82.7 | 74.7 | 54.0 | 52.1 | 51.2 | 61.7 | 17.9 | 30.5 | 48.7 | 20.5 | 49.4 |\n| DWA  \\[E\\]     | 81.0  | 72.3 | 50.2 | 50.1 | 49.0  | 63.4 | 10.7 | 34.3 | 32.9 | 6.9  | 44.6 |\n| PGT-Aug       | 83.0 | 71.8 | 54.8 | 51.1 | 54.9 | 69.7 | 20.2 | 39.5 | 49.6 | 14.5 | 50.9 |\n| PGT Aug \\+ CBLoss  | 82.7 | 74.7 | 56.5 | 55.9 | 54.4 | 68.7 | 20.9 | 34.1 | 53.5 | 20.5 | 52.2 |\n\n[D] Class-Balanced Loss Based on Effective Number of Samples (CVPR 2019)\\\n[E] Resolving Class Imbalance for LiDAR-based Object Detector by Dynamic Weight Average and Contextual Ground Truth Sampling (WACV 2023)\n\n---\n**[W2, Q1] The motivation of this paper**\nIn 3D object detection, there have been many studies aimed at addressing class imbalance problems by modifying model structures (CBGS) or adjusting loss function (DWA). The most widely used method is the over-sampling known as GT-Aug, which involves inserting objects from other frames into the current frame. However, this method is limited to inserting objects from a predefined pool (training set), which restricts learning intra-class diversity, and generating or collecting 3D data for various objects has been extremely challenging and expensive. Recent advancements in differentiable 3D reconstruction techniques, such as NeRF and Gaussian Splatting, have made it possible to reconstruct dense 3D points at a lower cost. By leveraging these 3D reconstruction techniques, we proposed a novel and practical pipeline that overcomes the limitations of traditional insertion methods, particularly in terms of cost and diversity.\n\n---\n**[W3] The detail of the framework**\nDue to page limits, we have elaborated the details of our Intensity Domain Alignment network in line 559 of supplementary material. In summary, we adopt CycleGAN framework to learn unpaired translation between RGB and intensity. We designed generators and discriminators with PointNeXt encoders. It receives nuScenes long-tail class samples as real data, and we create fake data by translating, rotating, resizing, and projecting long-tail class RGB samples to the same (x,y,z,l,w,h, theta) of real data. Our generator is trained to generate fake intensity values from RGB features, while discriminator is trained to discriminate between real and fake intensity.\n\n---\n**[W4] The effect of PGT-Aug on the majority of classes**\nOur primary objective is to generate and insert the minority classes instead of the majority classes to relieve the class imbalance issue. Thus, we anticipated that the performance of the minority would improve while the performance of the majority classes would either remain stable or improve slightly due to the synergy effect of addressing the imbalance. To verify the effectiveness of the pipeline to majority classes, we conducted experiments on KITTI dataset as shown in Table VI. We reconstructed 10 cars of 3D RGB point clouds from CO3D dataset [I], (See Figure III in Rebuttal-Supp.) and created about 16,000 pseudo LiDARs of Car class. We apply pseudo LiDARs along with GT LiDARs during augmentation, and our PGT-Aug performance largely outperforms GT-Aug, Real-Aug, and other augmentation methods in car detection benchmark. Due to writing limit, please refer to Experiment Details (jNRg).\n\n**Table VI. The effect of PGT-Aug on the majority class**\n|  | AP\\_car 70 (40 recall) |  |  |\n| ---- | :---: | :---: | :---: |\n|  | Easy | Mod | Hard |\n| Baseline | 88.08 | 74.85 | 70.55 |\n| GT-Aug | 87.80 | 78.36 | 75.41 |\n| LiDAR-Aug \\[G\\] | 87.75 | 78.24 | 75.35 |\n| 3D-VField \\[H\\] | 87.05 | 77.13 | 75.55 |\n| CA-Aug \\[F\\] | 88.82 | 78.66 | 75.75 |\n| Real-AUG (Our impl.) | 88.13 | 78.97 | 76.06 |\n| PGT-AUG | 90.00 | 79.45 | 76.35 |\n\n[F] Context-Aware Data Augmentation for LiDAR 3D Object Detection (ICIP 2023)\\\n[G] LiDAR-Aug: A General Rendering-based Augmentation Framework for 3D Object Detection (CVPR 2021)\\\n[H] 3D-VField: Adversarial augmentation of point clouds for domain generalization in 3d object detection (CVPR 2022)\\\n[I] Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (ICCV 2021)\n\n---\n**[Q2] Memory usage and inference time**\nIf Object-level Domain Alignment is performed during training of detection, it may take additional time compared to Real-Aug. However, in 3D object detection methods, objects to be inserted were stored before training, and objects were loaded and inserted during training. Loading these objects from disk to memory in order to insert them into the scene takes time, and the memory and time complexity for the batch is O(n). Therefore, the time cost is the same as Real-Aug.  Even though real-time performance was not a major consideration, we will add this discussion to the paper according to the valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K6AXdYFT4s", "reviewer": "Reviewer_jNRg", "summary": "This paper proposes a new method to augment pseudo-LiDAR point cloud to resolve the class-imbalance problem. It consists of generated 3D models of minority classes from video or miniaturized models with 3D reconstruction NERFs. Such models are then sampled from the target LiDAR and added to a Real LiDAR point cloud. LiDAR intensity are generated using a CycleGAN. Experiments have been conducted on several Dataset: KITTI, nuScenes and Lyft. It shows that the proposed method is competitive with the state-of-the-art and outperforms on minority classes.", "questions": "An ablation study is proposed with several experiments. It is interesting to see thant using 3DGS instead of Nerf increases the data augmentation process.\nIt could be interesting to see the impact of each step of the pipeline on the final performance by replacing each step by a simpler one. What happen if the luminance generation part is removed or replace by a simpler method? What happen if the 3D reconstruction step is produced with less images? What happen if the object level data alignment is removed and replaced by a general alignment?", "limitations": "/", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easy to follow. The related works section is well written and provides a good overview of the state-of-the-art methods. However, I suggest adding one general paragraph on Data Augmentation and it mains techniques. The proposed model combined several high level techniques such as 3D reconstruction and CycleGAN. The overall architecture is well explained and the intuition behind the method is clear. Due to the limited space, some details are given in the supplementary material.\n\nThe experimental part is two-fold: 1) the comparison with the state-of-the-art methods, and 2) the ablation study. The proposed model increases GT-Aug performances about 1% on both mAP and NDS. The improvment remains for several models. It demonstrates the genericity of the proposed augmentation pipeline. More SOTA models", "weaknesses": "Regarding the comparison with the state-of-the-art methods, the proposed method is evaluated on several datasets and compared with two data augmentation methods: GT-Aug (vanilla synthesis-based LiDAR data augmentation) and Real-Aug. GT-Aug is more a baseline than a competitor and to my point of view, the only competitor is Real-Aug in this experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n---\n**[S1] Additional Paragraph for Related Work**\nThis is a great suggestion. We will add more paragraphs to the Related Work Section, summarizing the literature on data augmentation techniques.\n\n---\n**[W1] Additional Comparison against Other Approaches**\n This is a good question. We chose Real-Aug as our main competitor because it is the latest work and shows promising scores in the public nuScenes leaderboard. However, we fully agree with the reviewer that additional comparison with other augmentation techniques is needed. Thus, as shown in Table VII below, we conduct experiments to analyze the data augmentation effect with other recent approaches, including LiDAR-Aug [F], CA-Aug [G], and 3D-VField [H]. Our experiment confirms that all approaches help improve detection accuracy against the baseline. Notably, Real-Aug and (our proposed) PGT-Aug generally outperform the other approaches. We further provide qualitative analysis in Figure III in our rebuttal pdf file.\n\n**Table VII. Performance comparison with other data augmentation approaches.**\n|  | AP\\_car 70 (40 recall) |  |  |\n| ----- | :---: | :---: | :---: |\n|  | Easy | Mod | Hard |\n| Baseline | 88.08 | 74.85 | 70.55 |\n| GT-Aug | 87.80 | 78.36 | 75.41 |\n| LiDAR-Aug \\[G\\] | 87.75 | 78.24 | 75.35 |\n| 3D-VField \\[H\\] | 87.05 | 77.13 | 75.55 |\n| CA-Aug \\[F\\] | 88.82 | 78.66 | 75.75 |\n| Real-Aug (Our impl.) | 88.13 | 78.97 | 76.06 |\n| PGT-Aug | 90.00 | 79.45 | 76.35 |\n\n**Experiment Details**\nWe compare PGT-Aug with other 3D data augmentation methods on the public KITTI [8] Car benchmark. We create pseudo LiDAR point clouds for our method by randomly sampling ten cars from the CO3D dataset. The instances used are as follows: 106_12650_23736, 106_12662_23043, 157_17286_33548, 185_19982_37678, 194_20899_41094, 216_22796_47484, 216_22827_48422, 421_58405_112551, 206_21799_45886, and 421_58407_112553.\n\n[F] Context-Aware Data Augmentation for LiDAR 3D Object Detection (ICIP 2023)\\\n[G] LiDAR-Aug: A General Rendering-based Augmentation Framework for 3D Object Detection (CVPR 2021)\\\n[H] 3D-VField: Adversarial augmentation of point clouds for domain generalization in 3d object detection (CVPR 2022)\n\n---\n**[Q1] More Ablation Studies**\n We conduct the following additional experiments: 1) without luminance generation (constant intensity), 2) instance generation according to the change in the number of images, and 3) without object level data alignment (random sampling, without rigid motion model). On the second row of Table VIII, the removal of luminance (intensity) generation in pseudo LiDAR leads to performance drop in downstream detection tasks. Also, as the percentage of multiview images used during 3D reconstruction decreases, the overall detection performance decreases accordingly. Finally, we conduct experiments on general alignment by replacing object-alignment with random sampling points from dense RGB point clouds and removing motion models. Both experiments show suboptimal performance compared to PGT-Aug.\n\n**Table VIII. Ablations in intensity, the number of images, data alignment**\n|  | Car  | Ped | Barrier  | T.C.  | Bus  | C.V. | Trailer  | Truck  | Motor  | Bicycle  | mAP | NDS |\n| :---- | :-----: | :-----: | :-----: | :-----: |:-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n| Ours | 85.4 | 85.4 | 68.0 | 71.1 | 72.1  | 24.2 | 40.4  | 59.8  | 70.3  | 58.3  | 63.52  | 69.11 |\n| constant intensity | 85.4 | 85.0 | 68.1 | 71.5 | 72.6 | 23.6 | 38.9 | 60.0 | 68.3 | 54.2 | 62.75 | 68.75 |\n| 25% of the number of image | 85.3 | 85.0 | 67.3 | 71.1 | 73.4 | 23.1 | 40.1 | 59.2 | 70.2 | 56.4 | 63.12 | 68.78 |\n| 50% of the number of image | 85.5 | 84.9 | 68.5 | 71.4 | 72.0 | 23.1 | 41.4 | 59.8 | 69.5 | 56.4 | 63.26 | 69.03 |\n| no-motion | 85.4 | 85.2 | 67.8 | 71.5 | 71.9 | 23.7 | 41.0 | 60.1 | 69.0 | 56.6 | 63.22 | 68.80|\n| random sampling | 85.4 | 85.1 | 67.2 | 71.4 | 73.2 | 22.8 | 39.2 | 59.8 | 68.1 | 55.4 | 62.76 | 68.56 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their time and their thoughtful comments and questions. We are encouraged that the reviewers find that: “very well-written and easy to follow” (R-iAKq, R-eXGm, R-kswe, R-jNRg); our method being described as “novel, cost-effective pipeline” (R-iAKq), “superiority and generality” (R-kswe), “well explained and a clear intuition behind the method” (R- jNRg), and “thorough methodology” (R-iAKq); our experiments were commended as “ample” (R-eXGm), “extensive” (R-kswe), “good” (R-kswe), backed by “robust evaluations” (R-iAKq), “intuitive visualizations” (R-kswe), demonstrating “genericity” (R-jNRg) and “effectiveness” (R-kswe).

We attempted our best to address the questions as time allowed. We believe the comments and revisions have strengthened the paper, and we thank all the reviewers for their help. Please find individual responses to your questions below.

Summary of major changes:
1. We add new experiments on KITTI [8] and nuScenes [9] to demonstrate its superiority compared to other augmentation methods and the effectiveness to solve class imbalance problems, respectively.
2. We provide additional ablation studies to show the impact of each individual model on the proposed pipeline.
3. We add detailed visualizations and explanations to highlight our contributions and facilitate a clear understanding of the proposed method.

Meta-review: This paper received mixed ratings of two Weak Accepts, Borderline Reject and Borderline Accept.  Three Reviewers sharing positive opinions and see the merits of this work.  Reviewer kswe (who suggested Weak Reject) has concerns in the comparative experiments, motivation, framework details, and the effect on the majority of classes. The authors provided detailed responses during the Rebuttal phase, while Reviewer kswe did not response. After reading the authors rebuttal and the reviewers' comments, the AC believes the concerns of Reviewer kswe have been mainly addressed. The AC agrees with the other 3 reviewers and suggests Accept. The authors are suggested to includes these importants results provided in Rebuttal into the camera ready version.
Decision: Accept (poster)