3D Copy-Paste: Physically Plausible Object Insertion for Monocular 3D Detection
Authors: Yunhao Ge
Hong-Xing Yu
Cheng Zhao
Yuliang Guo
Xinyu Huang
Liu Ren
Laurent Itti
Jiajun Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D Copy-Paste
Object Insertion
Monocular 3D Object Detection
Physically Plausible
Data Generation
TL;DR: 3D Copy-Paste automatically copies virtual objects and pastes them into a real scene. The resulting objects in the scene have 3D bounding boxes with plausible physical locations and appearances.

Abstract: A major challenge in monocular 3D object detection is the limited diversity and quantity of objects in real datasets. While augmenting real scenes with virtual objects holds promise to improve both the diversity and quantity of the objects, it remains elusive due to the lack of an effective 3D object insertion method in complex real captured scenes. In this work, we study augmenting complex real indoor scenes with virtual objects for monocular 3D object detection. The main challenge is to automatically identify plausible physical properties for virtual assets (e.g., locations, appearances, sizes, etc.) in cluttered real scenes. To address this challenge, we propose a physically plausible indoor 3D object insertion approach to automatically copy virtual objects and paste them into real scenes. The resulting objects in scenes have 3D bounding boxes with plausible physical locations and appearances. In particular, our method first identifies physically feasible locations and poses for the inserted objects to prevent collisions with the existing room layout. Subsequently, it estimates spatially-varying illumination for the insertion location, enabling the immersive blending of the virtual objects into the original scene with plausible appearances and cast shadows. We show that our augmentation method significantly improves existing monocular 3D object models and achieves state-of-the-art performance. For the first time, we demonstrate that a physically plausible 3D object insertion, serving as a generative data augmentation technique, can lead to significant improvements for discriminative downstream tasks such as monocular 3D object detection. Project website: https://gyhandy.github.io/3D-Copy-Paste/.

3D Copy-Paste: Physically Plausible Object Insertion for Monocular 3D Detection
Yunhao G Hong-Xing Yu', Cheng Zhao², Yuliang Guo Xinyu Huang Liu Ren", Laurent Itti", Jiajun Wu
Stannford University UUniversity of Southern California
Boosh Research North America, Bosch Center for Artificial Intelligence (BCAI) (yunhaoge, koven, jiajumnu)ccs. stanford edu (yunhaoge, itti]Qusc. edu (Cheng.Zhar Zhao, Yuliang.Guo2, Xinyu. Huang, Liu Liu.Ren) hen)eussbbschh.... bosch. com
Abstract
A major challenge in monocular 3D object detection is the limited diversity and quantity of objects in real datasets. While augmenting real scenes with virtual objects holds promise to improve both the diversity and quantity of the objects, it remains elusive due to the lack of an effective 3D object insertion method complex real captured scenes. In this work, study augmenting complex real indoor scenes with virtual objects for monocular 3D object detection. The main challenge is to automatically identify plausible physical properties for virtual assets (e.g., locations, appearances, sizes, etc.) in cluttered real scenes. To address this challenge. we propose physically plausible indoor 3D object insertion approach to automatically copy virtual objects and paste them into real scenes. The resulting objects in scenes have 3D bounding boxes with plausible physical locations and appearances. In particular, our method first identifies physically feasible locations and poses for the inserted objects prevent collisions with the existing room layout. Subsequently, it estimates spatially-varying illumination for the insertion location, enabling the immersive blending of the virtual objects into the original scene with plausible appearances and cast shadows. We show that our augmentation method significantly improves existing monocular 3D object models and achieves state-of-the performance. For the first time, we demonstrate that a physically plausible 3D object insertion, serving as generative data augmentation technique, can lead to significant improvements for discriminative downstream tasks such as monocular 3D object detection. Project website; https: /// //gh/ndyyyyy... wawaaanddggggttiit github Lop3D-Copy-aa Paste/
1 Introduction
Monocular indoor 3D object detection methods have shown promising results in various applications such as robotics and augmented reality (Yang and Scherer 2019 Chen et al. 2017). However, the deployment of these methods is potentially constrained by the limited diversity and quantity of objects existing real datasets. For example, in SUN RGB-D dataset (Song et al. 2015), the bathtub category has only less than 500 annotations compared to chair which has This may be due to the difficulty in acquiring and labeling substantial indoor scene datasets with
19,000 annotations.
diverse 3D object annotations [Silberman et al. 2012 Song et al. 2015 Dai al. 2017
Data augmentation techniques have been widely utilized 2D detection and segmentation tasks to improve the diversity and quantity of the available training data (Dwibedi et al. [017]] Ge et al.) 2022a Ghiasi et al.) 202T Ge al., 2022b 2023). However, it non-trivial to scale 2D augmentation methods to 3D scenes due to physical constraints in real 3D scenes. In particular, technical challenges
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Oijjects Objavame))
3D CopysPatts
Monocular
Object
Plassble
Detection
Indoor Scene Datanel (e.g. SUN RGB D)
40.55
Pontion
Size
InVoosefees
Figure 1: Overall pipeline of physically plausible object insertion for monocular 3D object detection: Our approach copies external 3D objects (e.g.. from Objaverse (Deitke al.) 2022]) and pastes them into indoor scene datasets (e.g.. SUN RGB-D [Song et al. 2015)) in physically plausible manner. The augmented indoor scene dataset, enriched with inserted 3D objects, then used train monocular 3D object detection models, resulting in significant performance improvements.
emerge especially in how to maintain physical plausibility for: (1) Collision and Occlusion Handling: In 3D data augmentation, handling collisions between objects is more challenging than in 2D data. Properly managing collisions is essential to prevent artifacts and ensure that objects appear as natural and coherent parts of the scene. (2) Illumination and Shading: For 3D data, augmenting objects requires careful consideration of the lighting conditions in the scene to create realistic shading and reflections. This involves estimating the spatially-varying illumination and adapting the appearance of the inserted objects to maintain visual coherence. (3) Geometric Consistency: In 3D data augmentation, maintaining geometric consistency is crucial to ensure that the augmented objects fit naturally within the scene. Unlike 2D augmentation, which deals with flat images, 3D augmentation must consider spatial relationships, object orientations, and their interaction with the surrounding environment.
In this paper, we explore novel approach, 3D Copy-Paster to achieve 3D data augmentation in indoor scenes. We employ physically plausible indoor 3D object insertion to automatically generate large-scale annotated 3D objects with both plausible physical location and illumination. Unlike outdoor scenarios, indoor environments present unique challenges: (1) complex spatial layouts, notably cluttered backgrounds and limited space for object placement, which require meticulously crafted method for automated object positioning (ensuring realistic position, size, and pose), and (2) intricate lighting effects, such soft shadows, inter-reflections, and long-range light source dependency, which necessitate sophisticated lighting considerations for harmonious object insertion.
Fig. shows our overall pipeline. In approach, we take advantage of cxisting large-scale 3D object datasets, from which we copy simulated 3D objects and paste them into real scenes. To address the challenges associated with creating physically plausible insertions, First, we analyze the scene by identifying all suitable planes for 3D object insertion. Next, we estimate the object's pose and size, taking into account the insertion site to prevent collisions. Lastly, we estimate the spatially-varying illumination to render realistic shading and shadows for the inserted object, ensuring that it is seamlessly blended into the scene.
employ threc-step process.
Our proposed method augment existing indoor scene datasets, such as SUN RGB-D [Song et al.. 2015]. by incorporating large-scale 3D object datasets like Objaverse [Deitke et al. 2022 using our 3D Copy-Paste approach. Our method is an offline augmentation method that creates a new augmented dataset. The monocular 3D object detection model, ImvoxelNet Rukhovvch et al. (2022) trained on this augmented dataset, achieves new state-of-the-art performance on the challenging SUN RGB-D dataset. We systematically evaluate the influence of the inserted objects' physical position and illumination on the downstream performance of the final monocular 3D object detection model. Our results suggest that physically plausible 3D object insertion serve an effective generative data augmentation technique, leading to state-of-the-an performances in discriminative downstream tasks such monocular 3D object detection.
We make three main contributions: (1) We introduce 3D Copy-Paste, a novel physically plausible indoor object insertion technique for automatically generating large-scale annotated 3D objects. This

---

approach ensures the plausibility of the objects' physical location, size, pose, and illumination within the scene. (2) We demonstrate that training a monocular 3D object detection model on dataset augmented using our 3D Copy-Paste technique results in state-of-the-art performance. Our results show that a physically plausible 3D object insertion method can serve as an effective generative data augmentation technique, leading to significant improvements in discriminative downstream monocular 3D object detection tasks. (3) We conduct a systematic evaluation on the effect of location and illumination of the inserted objects on the performance of the downstream monocular 3D object detection model. This analysis provides valuable insights into the role of these factors in the overall effectiveness of our proposed approach.
2 Related Works
2.1 Monocular 3D Object Detection
Monocular 3D Object Detection estimates the 3D location, orientation, and dimensions (3D bounding box) of objects from a single 2D image. It has garnered significant attention in recent years due its potential applications in autonomous driving, robotics, and augmented reality. There are many works
of monocular 3D detection in driving scenarios, such as 3DOP(Chen et al., 2015]. MLFusion Xu and Chen. 2018). M3D-RPM RPN Brazil and Liu, 2019]] MonoDIS S et al. 2019]. Pseudo-
LiDAR (aang et al., 2 FCOS3D Wang et al.] [202T], SMOKE et al., (020], RTM3D[Li
CaDDN RRadding et al. 2021). Specifically, Geometry-based utilized both LiDAR -hased point clouds and geometric cues from images for 3D object detection. Mousavian et al. [017] introduced method that regresses object properties such as dimensions, orientation, and location from 2D bounding boxes using geometric constraints. In the context of indoor scenes, multi-task learning has gained traction. Recent studies, including PointFusion by Xu et al. 20188), have amalgamated 3D object detection with tasks like depth estimation semantic segmentation improve performance. Total3D [Nie et al.) 2020) and Implicit3D (Zhang et al., 2021] use end-to-end solutions to jointly reconstruct room layout, object bounding boxes and meshes from single image. ImvoxelNet (Rukhovich al.. [002 achieves state-of-the. the--h--trrrrrr art performance by using the image-voxels projection for monocular 3d object detection.
et al. 2020a]. PGD| Wang et al. 2022a
Approaches: MV3D Chen et al. 2017
2.2 3D Data Augmentation
Data augmentation in 3D has become increasingly vital for enhancing performance across various 3D perception tasks. Most of work focuses on outdoor scenes (Zhang et al., 2020 Lian et al.. Abu Alhaija et al.) 2018 [Chen et al.) 2021] Tong et al.] 2023]. Geometric Transformations: Wu et al. [0155] applied rotations, translations. and scaling to augment the ModelNet dataset, improving classification and retrieval tasks. Point Cloud Augmentation: EEgelekee et al. [017] proposed techniques such as random point removal, Gaussian noise, and point cloud interpolation for augmenting LiDAR datasets, enhancing object detection and segmentation performance. Generative Model-based Augmentation: [Smith and Meger [2017] used a conditional GAN to generate diverse and realistic 3D objects. Similarly, AAchiioptas al. [2018) employed VAE for learning a generative model of 3D shapes for shape completion and exploration tasks. However, while 3D generative models can achieve object-level augmentation, they are not scalable to scene-level augmentation. 2D generative models can produce highly realistic images, but they do not provide physically plausible 3D labels. 3D Common corruptions (Kar et al.) [022]] use 3D information to generate real-world corruptions for 2D dataset, which can evaluate the model robustness and be used as a data augmentation for model training, but does not support 3D) detection because it does not introduce new 3D object content.
Abu
2.3 Illumination Estimation
Illumination estimation is critical focus within computer vision research, given its crucial role in various applications. [Li et al. [2020b] addressed the inverse rendering problem for complex indoor scenes, estimating ppatiall--varying lighting, SVBRDF, and shape from a single image. Meanwhile, a differentiable ray tracing method combined with deep learning was proposed for the learning- based inverse rendering of indoor scenes [Zhu et al. 2022]. Additionally, research has been conducted on using deep learning for indoor lighting estimation, with methods like Deep Parametric Indoor Lighting Estimation offering enhanced accuracy and efficiency Gardner et al. 2019). Furthermore,
3

---

Compositional Image
Scane Image
WHAT Humination on the dbject
(4] Envirommnt
Lighsig
(e] Lightnn Extmation Regabration
Ubject
Insertion Render
put object
3D Bounding Box Label
Powtor,
Depth
Piane Reconstraction & Selection
(8) Insertion Parometer Search
Figure 2: 3D Copy-Paste method overview: Our method (a) processes the input RGB image and depth data to reconstruct floor planes that can accommodate inserted objects. (b) Using the reconstructed planes and information about objects in the original scene, we estimate physically plausible position, pose, and size for the inserted objects, ensuring they do not collide with existing objects. (c) We predict the opatially-varying lighting of the scene. (d) By registering the insertion position determined in (b) to centially-varyingg lighting, our light estimation module (d) refined an HDR environment map represent the lighting information for the inserted objects. (e) The insertion rendering module takes the position, pose, size, and lighting as input and inserts a 3D object into the real scene, adjusting the object's lighting and shadows accordingly ensure it seamlessly integrates natural and coherent part of the scene.
Wang et al. [2022b] introduced Neural Light Field Estimation, method that effectively models complex lighting conditions for virtual object insertion in street scenes. These studies underscore the potential of machine learning in improving illumination estimation capabilities rendering and computer vision tasks.
3
Methods
This section presents proposed physically plausible indoor 3D object insertion approach. Fig. shows our 3D Copy-Paste method overview. Section 3.1 addresses the question of "where and how to place the object", detailing the process of estimating suitable insertion positions, poses, and sizes for the objects while avoiding collisions with existing objects. Section explains "what illumination should we add the object": estimate the scene's spatially-varying illumination and render the inserted objects with realistic lighting and shadows. Section 3.3 describes how we create augmented dataset using the inserted objects and train monocular 3D object detection models.
3.1 Where and how: Physically Plausible Position, Pose, and Size Estimation
This section describes handling the first challenge of avoiding collisions during insertion by estimating physically plausible position, pose, and size parameters.
3.1.1 Ground Plane Selection
Given a scene and 3D object to insert, the initial question is where to place the object. To accommodate new object, must identify and understand the available regions where the object can be situated. We perform plane reconstruction to comprehend the scene's 's layout and subsequently, estimate physically plausible key parameters such position, size, and pose. Fig. 2 presents an overview of our plane reconstruction and selection module, which takes an RGB image and depth data as input and predicts all potential planes, then narrows down the ground plane.
To get a rough plane reconstruction, we followed the plane extraction method using Agglomerative Hierarchical Clustering (AHC) described in Feng et al. 2014). There three main steps: (1)

---

we construct graph with nodes and edges representing groups of points, obtained by dividing the point cloud (merging RGB with depth) into non-overlapping groups. (2) We then perform AHC on the organized graph to identify potential planes by merging nodes that belong to the same plane, continuing until the mean squared error of plane fitring surpasses a threshold. (3) We use a pixel- wise region-growing method to refine the detected planes. To further refine the extracted planes while preserving clear face textures and sharp features without losing geometric details, we utilize a back-end indoor plane optimization and reconstruction method described in Wang and Guo (2018) Specifically, we first partition the entire dense mesh into different planar clusters based on the planes extracted with AHC, treating them as plane primitives. We then create texture patch for each plane and sample points on it, followed by executing a global optimization process to maximize the photometric consistency of sampled points across frames by optimizing camera poses. plane parameters, and texture colors. Further, we optimize the mesh geometry by maximizing consistency between geometry and plane primitives, further preserving the original scene's sharp features, such as edges and corners of plane intersections. Finally, we get the reconstructed plane with the geometry parameters (e.g., surface normal).
To select a proper plane for insertion, we first identify all horizontal planes based on surface direction and the standard deviation along the Z-axis. Specifically, there are two constraints for considering plane as horizontal: (1) The plane must have a surface normal aligned with the positive direction of the Z-axis (opposite of the gravity vector), and (2) the standard deviation along the Z-axis should be smaller than predefined threshold. In our scenario, we aim to insert furniture into the scene, such as the ten interest classes in the SUN RGB-D dataset (Song al., 2015]: :: sofa, bed, chair, desk, table, nightstand, dresser, bookshelf, toilet, and bathtub. Consequently, we must identify the floor plane by selecting the horizontal plane with the lowest average Z value among all detected horizontal planes.
3.1.2 Constrained Insertion Parameter Search
To address the question of where and how to place the object, we estimate specific insertion parameters: position (p), size (s), and pose (o). We propose an efficient constrained insertion parameter searching algorithm to calculate plausible insertion parameters while avoiding collisions with existing objects in the scene (Algorithm D. Given the reconstructed floor plane, we first determine the search space for each parameter. For position, bounding box of the object and calculate the center of the bottom surface (p) as the optimization parameter of position. To prevent potential collisions between the inserted object and existing assets in the original scene, we search for a suitable position around the center of the reconstructed floor. As shown in Fig. (b).. we first calculate the floor's center < (Cz, Cy. Cz), and set search square, which uses twice the floor's s standard deviation along X axis, On. and Y axis, Ty. as square width and length. The insertion position sampled from a Uniform distribution inside the search square P> ~ Uf Cz az; Cr + .² and Py ~ U[,, oy. Cy + oy), <- (Pa.Py, Cz). For size (s), we use the height of the 3D bounding box of the object as the optimization parameter. For each object category, we first calculate the mean ma and standard deviation on of the height of the object belonging to the same category in the original scene dataset. We then assume the height size follows a Normal distribution and sample a height size from this Normal distribution: E (m,,,, oR). For the pose (o), we only allow the object to rotate along the Z-axis to maintain its stability. The optimization parameter is the rotation angles alone the Z-axis, which follows uniform distribution as o ~ 4[[-T, m].
want the inserted object to touch the floor, we find the 3D
Algorithm I details the Constrained Insertion Parameter Search algorithm. We first set search budget: k search iterations. For each iteration, we randomly sample each parameter (position, size, and pose) from their corresponding search spaces and calculate the inserted object's bounding box based on the sampled parameters. We then check for collisions with existing objects and quantitatively evaluate the degree of collisions. A direct approach for collision checking is to convert the inserted object into point cloud and then calculate the overlap with existing objects' point clouds. However, this method is time-consuming due the large number of points involved. We simplify the problem by converting the original 3D collision into 2D collision to speed up the collision check. Since the inserted objects are on the floor, if two objects collide, their 3D bounding box projections on the top view would also often collide (but not always, e.g., when an object may be placed under a table; we here ignore these candidate placements). In other words, we disregard the absolute value of the relative collision score. Utilizing an efficient collision check allows us to set relatively large search iteration number, such as k = 1000, while still maintaining limited search time (less than 0.5 seconds). We also consider a resize factor
3D volume and use the 2D collision projection
5

---

Algorithm 1: Constrained Insertion Parameter Search
Input: An RGBD image of the scene, reconstructed floor, 3D object belonging to the class of interest,
Output: Position (ú: 3D bounding box boltom center), size (8: 3D bounding box (bbox) height), and pose (ò: orientation along Z-axis)
Compute position search constrains: floor center e 4- (cz, Cy Cy, ca). standard deviation du and dy
Initialize search parameters: & < 1000, degree of collision + inf
for i (1. do
Sample position: ps, ~ U[c, ow. C.. a.] and Py U[c dy. + Fyl. p (pos Pys Ca)
Sample size: 8 (N(m, ( resize factor r ~ rl,, 8 s/r,
where my and 07 are mean and standard deviation of object height in class j in the raw dataset
Sample pose; ~ 2[[-T, ø] Calculate 3D bbox 23D, parameter based the sampled insertion parameter (p, and o)
Project 3D bbox 2D bbox zin in top view
Calculate collision score ((ZD)) with existing objects in scene
i == 0 then
| Return p, 8, 0
i < then
p p. s. 0
[ [+|
Return p. 6
to shrink the size of the inserted object to handle inserting large object in a small empty floor scenario. During the search, we terminate the process if we find an insertion with a collision score of O; otherwise, we continue to track the best insertion with the lowest collision score and return it after completing & search iterations.
3.2
What Illumination is on the object
3.2.1
Spatial-varying Illumination Estimation and Retrieval
To answer the question of what kind of illumination should be cast on the object, we first need to estimate the spatially-varying illumination of the scene. This process involves encapsulating intricate global interactions at each spatial location. To achieve this, we utilize the deep inverse rendering framework proposed by [i et al. 2020bb. Initially, we estimate intermediate geometric features such as albedo, normal, depth, and roughness. Subsequently, a LightNet structure, consisting of an encoder-decoder setup, ingests the raw image and the predicted intermediate features. This, turn, enables the estimation of spatially-varying lighting across the scene.
varying illumination is represented as environment maps. Specifically, each 4x4 pixel region in the raw image associated with an environment map, which captures the appearance of the surrounding environment and is used for reflection, refraction, or global illumination. These maps are spherical (equirectangular), representing the environment on a single 2D texture. The X-axis corresponds longitude, and the Y-axis corresponds to latitude. Each point on the texture corresponds to a specific latitude and longitude on sphere.
As depicted in Fig. 2 [c), the estimated
To obtain the environment map associated with the position of the inserted object, register and retrieve the corresponding environment map based on the estimated position after performing the constrained insertion parameter search.
3.2.2 Environment Map Refinement
Coordinate transformation. The environment map, estimated for the inserted object, is based on the local coordinates of the insertion position. In particular, it establishes coordinate system where the surface normal is designated as the Z-axis. In order to apply this map for relighting the inserted object using rendering method (such Blender), it becomes necessary transform the environment map to align with Blender's coordinate system.
Latitude completion. The estimated environment map only contains latitudes in the range (0, T/2) because the inverse rendering method cannot estimate the illumination beneath the surface, As shown in Fig, 2 d), complete the entire environment map by filling in artificial values in the second half.

---

Table 1: Statistics of external 3D objects from Objaverse
Deitke et al.
2022
Category
Bed
Table
Sofa
Chair
Desk
Dresser
Nightstand
Bookshelf
Toilet
Bathtub
Number
190
854
361
934
317
13
99
142
24
Intensity refinement. The estimated environment map is in Low Dynamic Range (LDR) format, lacking High Dynamic Range (HDR) details and high contrast. we use the predicted value directly, the rendered shadow appears relatively fuzzy. We refine the value by adjusting the scale in log space JIDR= = DDR where y is a hyperparameter.
to estimate the HDR value:
Finally, we input the HDR environment map after transformation and refinement, along with the position, size, and pose, into an insertion renderer (e.g.. Blender). This allows us obtain the inserted image with 3D bounding boxes serving as ground truth.
3.3 Dataset Augmentation with Insertion and Downstream Model Training
Given an indoor scene dataset and a set of interest classes C for potential insertion, we can identify external 3D objects set & that fall within these classes of interest. Before any insertion, we calculate the statistical parameters for each class of interest that we aim to augment. For every class j E C, we assume the size parameter (for instance, the height) fits Gaussian distribution. We then calculate the mean and standard deviation of this size parameter to guide the insertion of external objects. Here the detailed steps for insertion: For each scene within the indoor scene dataset, we randomly select a category j from the class of interest set C. Next, we randomly choose an instance from the external 3D objects set & that belongs plausible insertion method (Algorithm [ to integrate this external 3D object into the scene. We could train any downstream monocular 3D object detection model with the augmented dataset because we automatically obtain the 3D annotations of the inserted objects.
the selected class j. We then utilize our physically
4 Experiments
This section presents experiments to assess the effectiveness of our proposed object insertion method and evaluate how different insertion parameters affect the final performance of monocular 3D object detection.
3D
4.1 Dataset and Model Setting
Indoor scene dataset. We utilize the SUN RGB-D dataset [Song et al. 2015) as our primary resource for indoor scenes. It is one of the most challenging benchmarks in indoor scene understanding. SUN RGB-D comprises 10,335 RGB-D images captured using four distinct sensors. The dataset is divided into 5,285 training scenes and 5,050 test scenes. Furthermore, it includes 146,617 2D polygons and 58,657 3D bounding boxes, providing a comprchensive dataset for our research.
We also ScanNet dataset (Dai al. 2017]. ScanNet v2 is a large-scale RGB-D video dataset, which contains 1,201 videos/scenes in the training set and 312 scenes in the validation set. Adapting it for monocular 3D object detection, we utilized one RGB-D image per video, amounting to 1,201 RGB-D images for training and 312 for validation. We compute the ground truth 3D bounding box label for each of our used views from their provided scene level label, as some objects in the scene may not be visible in our monocular viewpoint.
External 3D object assets. The quality of 3D objects is crucial for effective insertion. Hence, Objaverse [Deitke et al. 2022). a robust dataset with over 800,000 annotated 3D objects. Using word parsing, extract objects that align with the classes of interest for monocular 3D object detection within SUN RGB-D. Table I shows the selected Objaverse data for each SUN RGB-D class.
Monocular 3D object detection model. We focus on the challenging task of monocular 3D object detection that relies solely on a single RGB image as input. We employ Im VoxelNet, which achieves state-of-the-art performance on the raw SUN RGB-D dataset using only single RGB image as input. Other existing methods either resort to using additional modalities and multiple datasets for extra supervision or exhibit underwhelming performance. For the purpose of monocular 3D object
7

---

Table 2: Im VoxelNet 3D monocular object detection performance on the SUN RGB-D dataset with different object insertion methods. When inserting randomly, the accuracy of the downstream object detector drops, i.e., the detector suffers from random insertions (which may have collisions, occlusions, incorrect lighting, etc.). In contrast, by only applying physically plausible position. size, and pose, performance significantly improved (41. 1.0%)... Further, when plausible lighting and shadows are added, our 3D copy-paste improves the accuracy of the downstream detector to new state-of-the-art accuracy (43.79%) We use mAP (%) with 0.25 IOU threshold.
Sctting
Insertion Position, Posc, Size
Insertion Illumination
mAP@0.25
Im VoxelNet
N/A
N/A
40.96
Im VoxelNet random insert
Random
Camera point light
37.02
Im VoxelNet 3D Copy- Paste (w/ light)
Plausible position, size, pose Plausible position, size, pose
Camera point light
41.80
Im VoxelNet 3D Copy-Paste
Plausible dynamic light
43.79
Table 3: Per class average precision (AP) of Im VoxelNet 3D monocular object detection performance on SUN RGB-D dataset.
Sctting
mAP00.25
bed
chair
sofa
table
bkshf
dosk
bathrub
toilet
dresser
nightstand
Im VexelNet VoxelNett+ Copy-Paste
40.96
72.0
55.6
53.0
41.1
7.6
21.5
29.6
76.7
19.0
33.4
43.79
72.6
57.1
24.1
80.7
22.3
36.9
detection, we train the same Im VoxelNet model on the original SUN RGB-D dataset and its various versions, each augmented via different insertion methods. All mAP results are mAP@0.25.
4.2 Physscally-plausible position, pose, size, and illumination leads to better monocular detection performance
Our 3D Copy-Paste focuses solving two challenges: (1) Where and how put the object: estimate the object's position, orientation, and size for insertion while ensuring no collisions. (2) What illumination on the object: we estimate the -aaring illumination and apply realistic lighting and shadows to the object rendering. The following experiments evaluate the model performance.
Table 2 presents the results of monocular 3D object detection the SUN RGB-D dataset, utilizing various object insertion augmentation techniques. The first row is the performance of Im VoxelNet trained on the raw SUN RGB-D dataset without any insertion. The "Im VoxelNet random insert" row displays results achieved through a naive 3D object insertion without applying physically plausible constraints (random location and Camera point light). This approach led to drop in accuracy from 40.96% to 37.02%, likely due to the lack of physical plausibility causing severe collisions and occlusions the final image. The "ImVexelNe" showcases the performance after implementing our method for only estimating physically plausible insertion position, pose, and size. Despite using a rudimentary camera point light, this approach outperforms "Im VoxelNett without any insertion, and also outperforms the naive "Im VoxelNet + random insert" (+4.78 improvement). This result shows that applying plausible geometry is essential for downstream tasks and makes 3D data augmentation useful over a naive, random augmentation. After further applying physically plausible dynamic light, our proposed "Im VIxelNel + 3D Copy-Paste" further improved the performance and achieved new slate-of-the-ar surpassing Im VoxelNet without insertion (+2.83 %) on monocular 3D object detection task. This performance improvement suggests that our 3D Copy-Paste insertion can serve an efficient data augmentation method to positively benefit downstream 3D object detection tasks. Table 3 shows detailed SUN RGB-D monocular 3D object detection results with Im VoxelNet each individual object category.
3D Copy-Paste (w/o light)" row
Table presents the results monocular 3D object detection on the ScanNet dataset. We utilized one RGB-D image per video: 1,201 for training and 312 for validation. We compute the ground truth 3D bounding box label for each of our used views from their provided scene-level label. For the baseline, we train an Im VoxelNet monocular 3D object detection model on the training set and test on the validation set. For our method, there are overlapping categories (sofa, bookshelf, chair, table, bed, desk, toilet, bathtub) in the 18 classes of ScanNet with our collected Objaverse data. We use our 3D Copy-Paste to augment the training set and train an Im VoxelNet model. All the training parameters the same the training SUN RGB-D dataset. We show the results on the average accuracy of

---

Table 4: Im 3D monocular object detection performance on the ScanNet dataset with different object insertion methods.
Setting
mAP
hed
chair
sofa
table
bkshf
desk
bathtub
toilet
NoxeleeNe
14.1
25.7
7.9
13.2
7.8
4.2
20.5
22.1
11.5
Im Voxeleeee 3D Copy-Paste
16.9
27.7
12.7
10.0
10.8
9.2
26.2
29.2
9.0
Table 5: Im VoxelNet 3D monocular object detection performance on SUN RGB-D dataset with different illumination during insertion rendering. All experiments the same Im VoxelNet model, insertion also uses our proposed physically plausible position, size, and pose.
Setting
Light source type
Intensity
Direction
With shadow?
mAP:0..25
Point Light] 1
Point
100W
Camera position
Yes
41,80
Point Light 2
Point
100W
Side (left)
Yes
42.38
Area Light
Area
100W
Camera position
Yes
42,67
Area Light
Area
100W
Side (left)
Yes
42.02
Spot Light]
Spot
100W
Camera position
Yes
40.92
Spot Light? 2
Spot
100W
Side (left)
Yes
42,10
Sun Light
Sun
Camera position
42.11
Sun Light 2
Side (left)
41.21
(Dynamic Light)
Estimated Plausible light
Dynamic Dynamic
Dynamic
41.83
(Dynamic Light)
Estimated Plausible light
Dynamic
43.79
the 8 overlapping classes (mAP@0.25) in the Table Our 3D Copy-Paste improves Im by 2.8% mAP.
4.3 Ablation study the influence of insertion illumination and position monocular 3D object detection
We first explore the influence illumination inserted objects on downstream monocular 3D object detection tasks. 5 shows the Im VoxelNet performance on SUN RGB-D with different illumination settings during 3D Copy-Paste. To eliminate the influence of other insertion parameters, we fix the estimated position, pose, and size for each scene among all experiments in Table
Fig. Big.BBprovver provides visualization of the effects of various light sources and light parameters during the insertion rendering process. The corresponding monocular 3D object detection results are presented Table 5 These illustrate how lighting not only impacts the visual perception of the inserted object from human observer's standpoint but also considerably affects the performance of downstream detection tasks. Thus, an accurate and physically plausible lighting estimation is crucial for both understanding the scene and for the practical application of downstream detection tasks.
Table. 2 shows the importance of physical position, pose, and size (local context) monocular 3D object detection tasks. We also explored the importance of the global context to the detection performance. The global context here whole scene. For instance, inserting a toilet into living room may not satisfy the global context. We propose plausible global context insertion method where the inserted object class considers the global scene information. Also, we could select an inserted class based on the floor size: insert larger size objects (e.g., bed, bookshelf) settings. We find considering the global context during the insertion on par with the random category selecting setting, and the following downstream detection model may not be sensitive that.
the semantic relationship of the inserted object the
only a large size floor. Table. 6 shows results on different
4.4 Qualitative Analysis
Fig. 4 shows the qualitative results of monocular 3D object detection on SUN RGB-D dataset. Our method demonstrates enhanced capabilities in detecting objects with significant occlusion, provides improved pose estimation, and effectively suppresses false positives.

---

Point light
Sun light
Spot light
Area light
Camera position
Physically plausible light
(wio shadow)
Camera position (wi shadow)
Left shadow)
Figure 3: Visualization of different illumination inserted objects.
Table 6: Ablation study of global context influence on Im Voxelleet monocular 3D object detection
performance SUN RGB-D.
Method
Follow global context?
Select class based empty size?
mAPQ0.25
Im VoxelNet + 3D Copy-Pash Popy-Paste
Yes
No
43.75
VoxelNett + 3D Copy-Paste
Yes
Yes
43.74
Im VoxelNet + Copy-Paste
No
Yes
42.50
Im VoxelNet + 3D Copy-Paste
No
No
43.79
ImVoxeiNet
ImVoxelNet
3D Copy-Paste
Figure 4: Qualitative results on the SUN RGB-D dataset.
Conclusion and Discussion
Our work addresses the challenge of scarce large-scale annotated datasets for monocular 3D object detection by proposing a physically plausible indoor 3D object insertion approach. This technique allows effectively augment existing indoor scene datasets, such as SUN RGB-D, with large- scale annotated 3D objects that have both plausible physical location and illumination. The resulting augmented dataset enables training monocular 3D object model that achieves new sute-of-the-aa performance. Our approach carefully considers physically feasible locations, sizes, and poses for inserted objects, avoiding collisions with the existing room layout, and estimates spatially-varying illumination to seamlessly integrate the objects into the original scene. We also systematically evaluate the impact of the physical position and illumination of the inserted objects on the performance of the final monocular 3D object detection model. This paper is the first to demonstrate that physically plausible 3D object insertion can serve as an effective generative data augmentation technique, leading to faate-of-the-ar performance in discriminative downstream tasks like monocular 3D object detection. Our findings highlight the potential of 3D data augmentation in improving the performance of 3D perception tasks, opening up new avenues for research and practical applications.
10

---

Acknowledgments. This work is in part supported by Bosch, Ford, ONR MURI N00014-22-1 2740, NSF CCRI #2120095. Amazon ML Ph.D. Fellowship. National Science Foundation (award 2318101), C-BRIC (one of six centers in JUMP, Semiconductor Research Corporation (SRC) program sponsored by DARPA) and the Army Research Office (W91 INF2020053). The authors affirm that the views expressed herein are solely their own, and do not represent the views of the United States government or any agency thereof.
References
H. Abu Alhaija, S. K. Mustikovela, L. Mescheder, A. Geiger, and C. Rother. Augmented reality meets computer vision: Efficient data generation for urban driving scenes. International Journal of Computer Vision, 126:961-972, 2018.
Achlioptas, O. Diamanti, I. Mitliagkas, and Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning. pages 40-49. PMLR, 2018.
G. Brazil and X. Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9287-9296. 2019.
X. Chen, K. Kundu, Y. Zhu, A. G. Berneshawi, H. Ma, S. Fidler, and Urtasun. 3d object proposals for accurate object class detection. Advances in neural information processing systems, 28, 2015.
X. Chen, H. Ma, Wan, B. Li, and T. Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 1907-1915, 2017.
Y. Chen, F. Rong. S. Duggal, S. Wang. X. Yan, S. Manivasagam, S. Xue, E. Yumer, and R. Urtasun. Geosim: Realistic video simulation via geometry-aware composition for self-driving. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pages 7230-7240, 2021.
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-588 2017.
M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051,, 2022.
D. Dwibedi, I. Misra, and M. Hebert. Cut, paste and learn: Surprisingly easy synthesis for instance detection. In Proceedings of the IEEE international conference on computer vision, pages 1301- 1310, 2017.
M. Engelcke, D. Rao, D. Z. Wang, C. H. Tong, and I. Posner. Vote3deep: Fast object detection in 3d point clouds using efficient convolutional neural networks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1355-1361. IEBE, 2017.
C. Feng, Y. Taguchi, and V. Kamat. Fast plane extraction in organized point clouds using agglomerative hierarchical clustering. In 2014 IEEE International Conference on Robotics and Automation (ICRA). pages 6218 6225.. IEEE, 2014.
M.-A. Gardner, Y. Hold-Geoffroy K. Sunkavalli, C. Gagné, and J.-F. Lalonde. Deep parametric the IEEE/CVF International Conference on Computer
indoor lighting estimation. In Proceedings the Vision, pages 71 7175-7183. 2019.
Y. Ge, H. Behl, J. Xu, Gunasekar, Joshi, Y. Song, X. Wang, L. Itti, and V. Vineet. Neural-sim: Learning to generate training data with nerf. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII, pages 477-493. Springer, 2022a.
11

---

Y. Ge, J. Xu, B. N. Zhao, L. Itti, and V. Vineet. Dall-e for detection: Language-drive driven context image synthesis for object detection. arXiv preprint 0et::2226.099992 2022b.
Y. Ge, J. Xu, B. N. Zhao, N. Joshi, L. Itti, and V. Vineet. Beyond generation: Harnessing text to image models for object detection and segmentation. arXiv preprini (xXi::3309005966 2023.
G. Ghiasi, Cui, A. Srinivas, R. Qian, T.-Y. Lin, E. D. Cubuk, Q. V. Le, and B. Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2918-2928, 2021.
O. F. Kar, T. Yeo, A. Atanov, and A. Zamir. 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18963-18974, 2022.
P. Li, H. Zhao, P. Liu, and F. Cao. Rtm3d: Real-time monocular 3d detection from object keypoints for autonomous driving. In European Conference on Computer Vision, pages 644-660 Springer, 2020a.
Li, M. Shafiei, Ramamoorthi, K. Sunkavalli, and Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2475-2484, 2020b.
Q Lian, B. Ye, R. Xu, W. Yao, and T. Zhang. Exploring geometric consistency for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1685-1699 2022.
Z. Liu, Z. Wu, and R. Tóth. Smoke: Single-stage monocular 3d object detection via keypoint estima- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 996-997, 2020.
A. Mousavian, Anguelov, J. Flynn, and Kosecka. 3d bounding box estimation using deep learning and geometry. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7074-7082, 2017.
Y. Nie, X. Han, S. Guo, Y. Zheng, J. Chang, and J. J. Zhang. TotalBdunndestanddin: Joint layout, object pose and mesh reconstruction for indoor scenes from single image. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 55-64, 2020.
C. Reading, A. Harakeh, J. Chae, and S. L. Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8555-8564 2021.
D. Rukhovich, A. Vorontsova, and Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general- purpose 3d object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2397-2406, 2022.
N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. ECCV (5), 7576:746-760 2012.
A. Simonelli, S. R. Bulo, L. Porzi, M. López- Sipez-Antequera, and P. Kontschieder. Disentangling monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1991-1999, 2019.
E. J. Smith and D. Meger. Improved adversarial systems for 3d object generation and reconstruction. In Conference on Robot Learning, pages 87-96. PMLR, 2017.
S. Song, S. P. Lichtenberg, and Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 567-576. 2015.
X. Sun, J. Wu, X. Zhang, Z. Zhang, C. Zhang, T. Xue, J. B. Tenenbaum, and W. T. Freeman. Pix3d: Dataset and methods for single-image shape modeling. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2974-2983, 2018.
12

---

W. Tong, J. Xie, T. Li, H. Deng, X. Geng, R. Zhou, D. Yang, B. Dai, L. Lu, and H. Li. 3d data augmentation for driving scenes on camera. arXiv preprint arXiv:2303.10340 2023.
C. Wang and X. Guo. Plane-based optimization of geometry and texture for rgb-d reconstruction of indoor scenes. In 2018 International Conference on 3D Vision (3DV), pages 533-541. IEEE, 2018.
T. Wang, X. Zhu, J. Pang, and D. Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 913-922, 922, 2021.
Wang. Z. Xinge, J. Pang, and Lin. Probabilistic and geometric depth: Detecting objects in perspective. In Conference on Robot Learning, pages 1475-1485, PMLR, 2022a.
Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 8445 8453, 2019.
Z. Wang. W. Chen, D. Acuma, J. Kautz, and S. Fidler. Neural light field estimation for street scenes with differentiable virtual object insertion. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II, pages 380-397. Springer, 2022b.
Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
B. Xu and Z. Chen. Multi-level fusion based 3d object detection from monocular images. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2345-2353, 2018.
D. Xu, D. Anguelov, and A. Jain. Pointfusion: Deep sensor fusion for bounding box estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2018.
S. Yang and S. Scherer. Cubeslam: Monocular 3-d object slam. IEEE Transactions on Robotics, 35 (4):925-938, 2019.
C. Zhang, Z. Cui, Y. Zhang, B. Zeng. M. Pollefeys, and S. Liu. Holistic 3d scene understanding from single image with implicit representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8833 8842, 2021.
W. Zhang, Wang, and C. C. Loy. Exploring data augmentation for multi-modality 3d object detection. arXiv preprint 2xXi:22022.12741 2020.
J. Zhu, F. Luan, Y. Huo, Z. Lin, Z. Zhong, D. Xi. R. Wang. H. Bao, Zheng, and R. Tang. Learning- based inverse rendering of complex indoor scenes with differentiable monte carlo raytracing. In SIGGRAPH Asia 2022 Conference Papers, pages 1-8, 2022.
13

---

A Experiments on more Monocular 3D Object Detection methods
2022 for monocular 3D object detec- tion. To show the robustness our 3D Copy-Paste across different downstream detection methods. We conducted additional experiments with another monocular 3D object detection model: Im- plictt3DUnderrtanddng (Im3D [Zhang et al. 202T)) The Im3D model predicts object 3D shapes, bounding boxes, and only the SUN RGB-D dataset but also the Pix3D dataset (Sun et al. 2018). which supplies 3D mesh supervision. The Im3D) training process consists of two stages. In stage one, individual modules the Layout Estimation Network, Object Detection Network, Local Implicit Embedding Network, and Scene Graph Convolutional Network are pretrained separately. In stage two, all these modules undergo joint training. We incorporate our 3D Copy-Paste method only during this second stage of joint training, and it's exclusively applied the 10 SUN RGB-D categories used in the main paper. We implemented our experiment following the official Im3D guidelines"
In our main paper, we utilize Im VoxelNet (Rukhovich et al.
layout within unified pipeline. Training this model necessitates not
Table 7 displays the Im3D results for monocular 3D object detection on the SUN RGB-D dataset, adhering to the same ten categories outlined in main paper. Im3D without insertion, attained a mean average precision (mAP) detection performance of 42.13% After applying our 3D Copy-Paste method-w which encompasses physically plausible insertion of position, pose, size, and light- the monocular 3D object detection mÅP performance increased to 43.34. These results further substanti- ate the robustness and effectiveness of our proposed method.
Table 7: Im3D Zhang et al. 2021] 3D monocular object detection performance on the SUN RGB-D dataset (same 10 categories as the main paper).
Setting
Insertion Position, Pose, Size
Insertion Illumination
mAP
Im3D
N/A Plausible position, size, pose
N/A
42.13
Im3D + 3D Copy-Paste
Plausible dynamic light
43.34
B
More experiment details
We run the same experiments multiple times with different random seeds. Table shows the main paper Table results with error range.
Table 8: Im VoxelNet 3D monocular object detection performance on the SUN RGB-D dataset with different object insertion methods (with error range).
Setting
Insertion Position, Pose, Size
Insertion Illumination
mAP@0.25
Im VoxelNeet
N/A
N/A
40.96 + 0.4
Im VoxelNet random insert
Random
Camera point light
37.02+ 0.4
VoxeINet 3D Copy-Puste (w/o light)
Plausible position, size, pose Plausible position, size, pose
Camera point light
41.80=t 0.3
Im VoxelNet 3D Copy-Paste
Plausible dynamic light
43.79 1 0.4
We also show our results with mAP@0.15 on SUN RGB-D dataset (Table 9) our method shows consistent improvements.
Table 9: Im VoxelNet 3D monocular object detection performance on the SUN RGB-D dataset with 15.@0.15.
Setting
Insertion Position, Pose. Size
Insertion Illumination
mAP@0.15
Voxel
N/A
N/A
48.45
Voxelleee + 3D Copy-Paste
Plausible position, size, pose
Plausible dynamic light
51.16
14

---

C Discussion on Limitations and Broader Impact
Limitations. Our method, while effective, does have certain limitations. A key constraint is its reliance on the availability of external 3D objects, particularly for uncommon categories where sufficient 3D assets may not be readily available. This limitation could potentially impact the performance of downstream tasks. Moreover, the quality of inserted objects can also affect the results. Possible strategies address this limitation could include leveraging techniques like Neural Radiance Fields (NeRF) to construct higher-quality 3D assets for different categories.
Broader Impact. Our proposed 3D Copy-Paste method demonstrate that physically plausible 3D object insertion can serve as an effective generative data augmentation technique, leading to state-of-the-ar performance discriminative downstream tasks like monocular 3D object detection. The implications of this work are profound for both the computer graphics and computer vision communities. From a graphics perspective, our method demonstrates that more accurate 3D property estimation, reconstruction, and inverse rendering techniques can generate more plausible 3D assets and better scene understanding. These assets not only look visually compelling but can also effectively contribute to downstream computer vision tasks. From computer vision perspective, it encourages us to utilize synthetic data more effectively to tackle challenges downstream fields, including computer vision and robotics.
15


Reviews:
[{"review_id": "3qZbhtdYDO", "reviewer": "Reviewer_7hAU", "summary": "The manuscript introduces system to create RGBD datasets augmented with additional 3d objects (one per rgb frame). The inserted objects are placed such that they dont intersect other objects, stand on the ground floor and the objects are rendered into the original RGB frame such that the lighting of the scene is respected. The augmented dataset can then be used to train slightly better 3d detectors since more 3d object examples can be supervised. ", "questions": "see weaknesses", "limitations": "Limitations such as the need for metrically scaled objects, the need for depth images and the knowledge of the gravity direction are not explicitly discussed in the paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The manuscript tackles an important problem for 3D object detection research: the scarcity of supervision data and the difficulty inherent in augmenting in 3D. The 3 factors for physical plausability make sense to me and the implementation of them in the paper seems pretty solid judging by the example images from the generated augmented dataset.\nThe claim that the full system of physically plausible augmentation is needed is well supported by ablations against other plausible choices (random placement, simpler light sources). The system seems sound although I am not an expert on the most recent ways of estimating illumination.\n", "weaknesses": "The primary weakness to my mind is the use of only a single (small!) 3d object detection dataset with SUB RGB - ScanNet is substantially bigger and commonly used. Who knows maybe this method would be even more powerful on a harder dataset?\n\nThe delta in mAP is not very big with the addition of the presented augmentation method (from 40.96 to 43.79). I question whether people will want to implement the full presented system in order for such modest gains. This limits the potential impact in my eyes and means claim (2) \"significant\" improvements is not fully supported.\n\nImVoxelnet is not a single-view detector (it is multiview but can be used for single-view detection); In the original paper, the mAP for SUN RGB-D is actually 43.66 (I suppose thats because of the different IoU threshold of 0.15 but it raises the question to me why the evaluation in this manuscript doesnt use the same one?)\n\nI am not 100% clear on the way the illumination is estimated and used as somebody who has never worked with such methods. This would make it hard for me to replicate the presented work.\n\nThere is a few sentences on semantic plausibility and that it didnt help but its not well explained how semantic plausibility was achieved. This is an important negative result that I am not sure I can trust at the current level of explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n- **Experiments on ScanNet**:\nWe use the SUN RGB-D dataset as this dataset offers 10,000+ monocular RGB-D images (scenes), and is densely annotated with 146,617 2D polygons and 58,657 3D bounding boxes. Many 3D object detection papers [1,2] use SUN RGB-D performance as the main result. ScanNet is a large-scale R-GBD video dataset that isn't specifically tailored for monocular 3D object detection. We appreciate the reviewer’s suggestion and have conducted new experiments on ScanNet.\n\n  Experimental settings are described in detail in the Global Response to all reviewers (above). We summarize it here:\n\n  [*Adapt to monocular setting*] We utilized one RGB-D image per video: 1,201 for training and 312 for validation. We calculate the ground truth 3D bounding box label for each of our used views from their provided video(scene) level label. \n\n  [*Training and test*] There are 8 overlap categories in ScanNet with our collected Objaverse data. We use our 3D Copy-Paste to augment the training set and train an ImVoxelNet. We show the average accuracy of the 8 overlap classes below. Our method improves ImVoxelNet by 2.8% mAP.\n\n|ScanNet AP_0.25| Average (mAP)|bed|chair|sofa|table|bookshelf|desk|bathtub|toilet|\n|-|-|-|-|-|-|-|-|-|-|\n|ImVoxelNet|14.1|25.7|7.9|**13.2**|7.8|4.2|20.5|22.1|**11.5**|\n|ImVoxelNet+3D Copy-Paste|**16.9**|**27.7**|**12.7**|10.0|**10.8**|**9.2**|**26.2**|**29.2**|9.0|\n\n- **Improvement significance and implementation easiness**:\nMonocular 3D object detection is a challenging task that requires inferring the 3D information given only a single RGB image, and improving from 40.96 to 43.79 can be considered significant as Reviewer DWmW also pointed out. If checking the performance leaderboard, it has been hard to improve mAP further beyond 40 (e.g., ImVoxelNet remains the current SOTA on Papers-With-Code on SUN RGB-D even though it is now 2 years old). Given our 2.83% improvement over that, our method is the new state-of-the-art performance on indoor scene monocular 3D object detection. \n\n  Beyond just a numerical improvement in mAP, our work provides a number of conceptual advances in the field, which we believe are important as well.  Firstly, the task of monocular 3D object detection is notably data-intensive, and labeling 3D labels can be both time-consuming and costly. Our approach addresses this challenge through data augmentation, introducing an automatic pipeline that remains model-agnostic. Since data augmentation is a one-off effort, it can potentially enhance various models. To allow easy usage, we will release our code, model, and generated data.\n  Secondly, through comprehensive experiments, we demonstrate the efficacy of our method across diverse models, such as ImVoxelNet and Implicit3D (detailed in the supplementary), and on different datasets, including SUN RGB-D and ScanNet. Importantly, our findings highlight the potential of 3D data augmentation in improving the performance of 3D perception tasks, opening up new avenues for research and practical applications.\n\n- **mAP 0.25 evaluation**:\nThis has to do with the fact that official results are only available for a few combinations of dataset/object classes/IOU threshold: The official ImVoxelNet GitHub code, when using SUN RBG-D on “10 classes from VoteNet” setting (same as ours), also uses 0.25 IOU threshold with performance 40.7 (even though they use 0.15 with other datasets/classes). \nThe authors provided an implementation in MMdetection3D Github [3], which also uses mAP 0.25 for the 10 classes from VoteNet setting (performance 40.96). We used the official code of MMdetection3D, so we used mAP 0.25. As you suggested, we also show our results with mAP 0.15 on SUN RGB-D dataset below. We will incorporate this in our revised paper.:\n\n|SUN RGB-D|mAP_0.15|\n|-|-|\n|ImVoxelNet|48.45|\n|ImVoxelNet + 3D Copy-Paste|**51.16**|\n\n- **Illumination estimation and reproducibility**:\nIllumination estimation is an important and challenging task, our main paper Sec. 2.3 (L108~L119) listed some representative works in this domain. We will release all our code and dataset upon acceptance for reproducibility.\n\n- **Semantic plausibility influence**:\nFor this experiment, to make the insertion more globally semantically plausible (e.g., avoid inserting toilet into other rooms than bathroom), we only insert the object categories that already exist in the current room. For instance, if the room contains table and chair, we only consider inserting new objects that belong to these 2 categories.\n\n  The results (Table 4) show that considering the global semantic meaning (43.75) is on par with the random category selecting setting (43.79). One potential reason is that the downstream detection (CNN-based models) may rely more on local information to conduct detection, so they are not sensitive to the global semantics. Different from point cloud-based 3D detection, where context information is important as RGB information is often discarded, in monocular 3D object detection where the input is an RGB image, appearance per se may be the most important clue. \n\n- **Requirement of depth, scale, and gravity**:\nThe metrically scaled object, depth, and gravity direction can be either dataset provided or estimated by off-the-shelf methods, e.g., metric depth estimation from ZoeDepth [4], and gravity direction estimation from ground segmentation with plane fitting.  We will add this discussion in the limitations section (currently, limitations are in the supplementary).\n\n**References**\n\n[1] Huang et al. \"Perspectivenet: 3d object detection from a single rgb image via perspective points.\" NeurIPS 2019.\n\n[2] Zhang et al. \"Holistic 3d scene understanding from a single image with implicit representation.\" CVPR 2021.\n\n[3] MMDetection3D: OpenMMLab next-generation platform for general 3D object detection.\n\n[4]\"Zoedepth: Zero-shot transfer by combining relative and metric depth.\" arXiv 2023."}, {"review_id": "g0hDcduHJ8", "reviewer": "Reviewer_sM4b", "summary": "This paper addresses the scarcity of large-scale annotated datasets which is challenging for rapid deployment of monocular 3D object detection models. A physically plausible indoor 3D object insertion approach is proposed to automatically copy and paste virtual objects into real scenes. The resulting objects have 3D bounding boxes with plausible physical locations and illumination, augmenting existing indoor scene datasets. The proposed data augmentation method achieves state-of-the-art performance in monocular 3D object detection. Most importantly, in this approach, a candidate selection process is applied along with a spatially varying illumination procedure from an existing method to ensure the plausibility of the objects’ physical location, size, pose, and illumination within the scene.  From the results, the location and illumination of the inserted objects appear to have a significant impact on the performance of the downstream model.", "questions": "I have a few questions that I hope the authors can help with. \n\n- First, I'm curious about what happens to out-of-distribution insertions such as the NYU dataset. Would it be helpful to include more datasets? \n- Second, I'm wondering how long it takes to render one object, taking into account the time it takes to search for the insertion point, correct the appearance, and other factors. \n- Lastly, I've noticed that most of the inserted objects are generally diffuse. I'm curious about how the object insertion process is affected when inserting shiny or specular objects. \n", "limitations": "Missing comparison with common 3D corruptions as data augmentation and comparing 2D recognition tasks + other 3D downstream tasks. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper's proposed approach is impressive. It automatically inserts virtual objects from ObjaVerse into real scenes, addressing the issue of limited annotated datasets in computer vision. The method ensures the objects' physical location, size, pose, and illumination, resulting in augmented indoor scene datasets. The use of plane selection methods, discarding objects with collisions using a simplified assumption, and constrained parameter search for insertion, along with the use of spatially-varying lighting estimation is well-thought-of and designed process. Its improved performance from resulting augmentations for monocular 3D object detection demonstrates the effectiveness of the method. Furthermore, the paper is very well-written and easy to follow. ", "weaknesses": "The paper lacks a comparison or discussion in relation to Common 3D Corruption (CVPR 2022; not cited). Even though common 3d corruption only evaluates 2D downstream tasks, it could still be utilized and compared for 3D object detection and demonstrate how important is 3D and illumination-aware physically grounded insertions. Another weakness is that the paper only evaluates one task. It would be beneficial to assess the impact of 3D grounded and illumination-aware object insertion on other 3D or 2D tasks. Could it also enhance 2D recognition tasks such as segmentation or detection? Including these aspects in the paper would reinforce its findings. It appears that the paper is lacking a basic 2D copy-paste baseline as well. It would be interesting to see how much this technique could potentially improve 3D tasks (Q: Is random insert in comparisons a 2D insertion or 3D insertion?). Additionally, the paper is missing ablation results for changes in object detection performance for each individual object.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n- **Comparison to Common 3D Corruption**:\n3D Common corruptions (3DCC) use 3D information to generate real-world corruptions, which can evaluate the model robustness and be used as a data augmentation for model training. Our method's contribution may be orthogonal to 3DCC. 3DCC conducted scene-level global augmentation and did not introduce new object content. Combining our method with 3DCC may achieve better performance. We will cite this paper and add comparisons in related works.\n\n\n- **Evaluation on other tasks**:\nThat is a very good point, we focus on monocular 3D object detection because it is a challenging, fundamentally representative, and data-intensive task, which involves both 3D geometry and semantic understanding, and our method could help. We also conducted experiments to show that the inserted position, size, pose, and lighting do influence the downstream model performance. We extend our method in other 3D detection datasets (ScanNet [1] below) and other models (Implicit3DUnderstanding [2] in supplementary), and we will treat extending to other tasks as future work. \n\n  While we agree with the reviewer that extending to other downstream tasks is desirable, this will take more time than available during the rebuttal period; in the meantime, we note that our title and overall positioning of the paper are not overselling our results, i.e., it is clearly stated that our method is useful for monocular 3D object detection. Likewise, we will clearly note in the manuscript that this is important but considered for future work beyond this paper.\n\n  Here are the detailed experiments and results on ScanNet:\n\n  [*Adapt to monocular setting*] ScanNet contains 1,201 videos(scene) in the training set and 312 videos(scene) in the validation set. Adapting it for monocular 3D object detection, we utilized one RGB-D image per video, amounting to 1,201 RGB-D images for training and 312 for validation. We calculate the ground truth 3D bounding box label for each of our used views from their provided video(scene) level label (because some objects in the scene may not be visible in our monocular view). \n\n  [*Training and test*] For the baseline, we train an ImVoxelNet monocular 3D object detection model on the training set and test on the validation set. For our method, there are 8 overlap categories (sofa, bookshelf, chair, table, bed, desk, toilet, bathtub) in the 18 classes of ScanNet with our collected Objaverse data (main paper Table1). We use our 3D copy-past to augment the training set and train an ImVoxelNet. All the training parameters are the same as the training on SUN RGB-D dataset. We will release all the code.\n\n  Table: ScanNet experiments.\n\n|ScanNet AP_0.25| Average|bed |chair|sofa|table|bookshelf|desk|bathtub|toilet|\n|-|-|-|-|-|-|-|-|-|-|\n|ImVoxelNet|14.1|25.7|7.9|**13.2**|7.8|4.2|20.5|22.1|**11.5**|\n|ImVoxelNet+3D Copy-Paste|**16.9**|**27.7**|**12.7**|10.0|**10.8**|**9.2**|**26.2**|**29.2**|9.0|\n\nFor the 2D recognition task, some related works [3,4] showed that simple 2D copy-paste is already good enough to help improve performance. While it is hard to conduct copy-paste in 3D, that is one of the motivations of our work. We believe our work should also help on 2D task and will treat it as future work.\n\n- **2D copy-paste baseline and individual object performance**:\n For 2D insertion, it is hard to obtain the 3D bounding box, which is required for downstream monocular 3D object detection.\nThe random insertion in the main paper comparison is 3D insertion, where the insertion position, size, pose, and illumination are not physically plausible, which causes a significant performance drop. \n\n  Here are the detailed SUN RGB-D monocular 3D object detection results of the main paper Table 2 on each individual object category:\n\n|SUN RGB-D AP_0.25 | Average (mAP)|bed |chair|sofa|table|book shelf|desk|bathtub|toilet|dresser|night stand|\n|--|--|---|---|--|--|--|--|--|--|--|--|\n| ImVoxelNet|40.96|72.0|55.6 |53.0|41.1|**7.6**|21.5|29.6|76.7|19.0|33.4|\n| ImVoxelNet+3D Copy-Paste|**43.79** |**72.6**|**57.1** |**55.1**|**41.8** |7.1|**24.1**|**40.2** |**80.7** |**22.3**|**36.9**|\n\n- **Include more datasets**:\n Yes, we posit that a richer collection of objects to insert would be beneficial. However, we need full 3D models that we can insert in any pose (thus, the NYU dataset may not work as it does not provide full 3D object models). But other 3D object datasets (e.g., OmniObject3D) could be included in future work.\n\n- **Time cost to render one object**:\n Overall it will take around 5~10 seconds. Specifically, searching the insertion position, pose, and size takes less than 0.5 seconds with iteration 1000. Plane detection, lighting estimation, and rendering take most of the time.\n \n- **Insert shiny or specular objects**:\nChanging the reflection property only influences the final rendering process; our physically plausible position, pose, size, and illumination are agnostic to the object's surface property. Specular objects will show reflections of other objects or lights during the rendering process.\n\n**References**\n\n[1] Dai, Angela, et al. \"Scannet: Richly-annotated 3d reconstructions of indoor scenes.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\n[2] Zhang, Cheng, et al. \"Holistic 3d scene understanding from a single image with implicit representation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[3] Dwibedi, Debidatta, Ishan Misra, and Martial Hebert. \"Cut, paste and learn: Surprisingly easy synthesis for instance detection.\" Proceedings of the IEEE international conference on computer vision. 2017.\n\n[4] Ghiasi, Golnaz, et al. \"Simple copy-paste is a strong data augmentation method for instance segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021."}, {"review_id": "BxrcPriuvn", "reviewer": "Reviewer_DWmW", "summary": "This paper presents a novel 3D augmentation method to augment the variety of 3D scenes with the corresponding 2D images. This augmentation method focuses on addressing the data-hungry problem when doing monocular 3D detection. The strategy here is intuitive: to insert 3D synthetic object models into 3D real scenes to augment the 3D scene data, and make sure consistent illumination, shading, and layout reasonability without any object collision issues. Therefore, the pipeline in this paper consists of three parts to answer three questions: 1. where and how to place the object in a 3D scene; 2. what is the illumination and how to add it onto object; 3. use the augmented dataset for network training.\n\nIn my view, the major contribution of this paper is the pipeline or concept: to leverage inverse rendering and re-rendering to augment 3D data and its corresponding 2D image for monocular 3D detection. Each module used in the pipeline already exists.\n\nThe experiments in this paper are pretty extensive, and this paper is well written.\n\n", "questions": "1. I wonder how to choose the object class category for insertion. Is it randomly sampled from an object class set? or manually? Because for indoor scenes, there is a strong scene context, e.g., it is not much likely to insert a \"bed\" in a \"toilet\". How do you consider such consistency?", "limitations": "The authors discussed the limitation and societal impact in the supplemental.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "As discussed above, the major strength of this paper is its concept and the tailored pipeline.\n\n1. This paper proposed a new strategy to do 2D-3D data augmentation using neural inverse rendering and re-rendering.\n\n2. The tailored pipeline successfully verified the idea, that such a data augmentation strategy can improve monocular 3D detection by a large margin.\n", "weaknesses": "In my view, the weakness is from the method contribution side:\n\n1. Neural inverse rendering to support object editing and augmenting is not novel. There are many works in image-based rendering and decomposition that can support inserting new objects into a 3D scene. I understand that the contribution here is to use it for data augmentation to support monocular 3D detection. But the methodology here is not novel.\n\n2. Many modules are from existing works (e.g., the inverse rendering framework, plane extraction) which makes this paper more like a novel combination to improve an existing task. I like the insight here by leveraging inverse rendering to do data augmentation for monocular 3D detection, and the performance is quite good. It would be more convincing if it also works for other datasets, e.g., ARKitScenes. Because in my view, the experiment performance is the other major contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n- **Methodology contribution**:\nTo conduct 3D object insertion for data augmentation, traditional methods[1] require manually locating support planes, designing poses, estimating lighting, etc, which are hard to scale up. Out method is the first automated 3D insertion pipeline on complex indoor scenes, enabling large-scale 3D insertion for data augmentation.\n\n  To allow a fully automated pipeline, we make the following technical contributions: (1) To avoid collision, after plane detection, we propose a constrained insertion parameter search method (algorithm 1) to guarantee a physically-plausible inserted position, pose, and size. We also include an efficient collision check method to solve the time-consuming challenge.  (2) We conduct environment map transformation and refinement to add more accurate illumination on inserted objects.\n\n  To our best knowledge, our work is the first to showcase that physically plausible 3D object insertion can serve as an effective generative data augmentation technique for indoor scenes, leading to state-of-the-art performance in discriminative downstream tasks such as monocular 3D object detection, opening up new avenues for research and practical applications.\n\n\n\n- **Experiments on other datasets**:\nThank you for your suggestion. We conduct new experiments on the ScanNet [2] dataset. Here are the detailed setting and results:\n\n  [*Adapt to monocular setting*] ScanNet contains 1,201 videos(scene) in the training set and 312 videos(scene) in the validation set. For monocular 3D object detection, we use one RGB-D image per video, so 1,201 RGB-D images for training and 312 for validation (test). We calculate the ground truth 3D bounding box label for each of our used views from their provided video(scene) level label (because some objects in the scene may not be visible in our monocular view). \n\n  [*Training and test*] For the baseline, we train an ImVoxelNet monocular 3D object detection model on the training set and test on the validation set. For our method, there are 8 overlap categories (sofa, bookshelf, chair, table, bed, desk, toilet, bathtub) in the 18 classes of ScanNet with our collected Objaverse data (main paper Table1). We use our 3D copy-past to augment the training set and train an ImVoxelNet. All the training parameters are the same as the training on SUN RGB-D dataset. We will release all the code.\n\n  We show the results on the average accuracy of the 8 overlap classes (AP_0.25) in the Table below. Our 3D Copy-Paste improves ImVoxelNet by 2.8% mAP.\n\n|ScanNet AP_0.25                   | Average (mAP)|bed |chair|sofa|table|bookshelf|desk|bathtub|toilet|\n|-----------------|------|------|----|----|----|----|----|----|----|\n| ImVoxelNet              |14.1 |25.7|7.9  |**13.2**|7.8  |4.2      |20.5|22.1   |**11.5**  |\n| ImVoxelNet+3D Copy-Paste|**16.9** |**27.7**|**12.7** |10.0|**10.8** |**9.2**      |**26.2**|**29.2**   |9.0   |\n\n\n\n- **Inserted object class selection**:\nGood point! The main results in Tables 2 and 3 use a random sample from the object class set, taken uniformly and without consideration of context. However, we also explore the influence of global context on detection performance in the main paper, Table 4. For this experiment, we only insert the object categories already existing in the current room to make the insertion more globally semantically plausible (e.g., avoid inserting toilets into other rooms except for the bathroom). For instance, if the room contains a table, chair, and sofa, we only consider inserting new objects that belong to these 3 categories. \n\n  The results (Table 4) show that considering the global semantic meaning (43.75) is on par with the random category selecting setting (43.79). One potential reason is that the downstream detection (CNN-based models) may rely more on local information to conduct detection, so they are not sensitive to the global semantics. Different from point cloud-based 3D detection, where context information is important as RGB information is often discarded, in monocular 3D object detection where the input is an RGB image, appearance per se may be the most important clue.\n\n\n**References**\n\n[1] Debevec, P., 2008. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Acm siggraph 2008 classes (pp. 1-10).\n\n[2] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T. and Nießner, M., 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5828-5839)."}, {"review_id": "DmwbX79gS1", "reviewer": "Reviewer_B4nT", "summary": "This work introduces 3D Copy-Paste, a  physically plausible indoor object insertion technique for automatically generating large-scale annotated 3D objects. This approach ensures the plausibility of the objects’ physical location, size, pose, and illumination within the scene\nUsing this 3D copy-paste as augmentation, a better monocular 3D detector can be trained.\nExperiments are demonstrate on the SUN RGB-D dataset, demonstrating the effectiveness of the proposed object insertion method in improving the 3D detection.\n", "questions": "I'm not familiar with the ImVoxelNet,I just quickly went through it to review this paper.  ImVoxelNet reported AP@0.15. Why AP@0.25 is reported in this paper?\n\nI will raise my rating if the authors can apply this approach to outdoor scenes  like kitti and nuscenes  and improve ImVoxelNet results.(ImVoxelNet is already tested on this dataset)\n", "limitations": "Apparently, the authors did not go over https://neurips.cc/public/guides/PaperChecklist.\nNot many experiments and implemented details are provided, Licenses and Assets  are not clearly described either.\nBut I don't penalize it in the rating though.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is clearly explained\n2. The proposed methods for object insertion can help create annotated data for free. \n3. The inserted object is physically plausible in location, size and illumination. Moreover, it's fully automated.\n4. Solid experiments are conducted in SUN RGB-D based on a SOTA detector (ImVoxelNet), and the detection AP is improved using the proposed method as augmentation. (40.96-->43.79)", "weaknesses": "This work may somewhat lack novelty. I'm not familiar with the indoor scene object insertion task, but for the outdoor scene, there are quite a lot similar works[1-4].  It would be better to include these works in the literature review. These works share similar ideas with the authors, where they insert object in physically plausible location and illumination. And some works also test with downstream tasks and demonstrate effectiveness of using actor insertion as data augmentation.    Besides,  I'm not fully convinced by the claim that indoor scenes are \"more challenging\" in L51. The layout, illumination, complexity of outdoor scenes are apparently more complex.\n\n[1] Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes\n[2] GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving\n[3] Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion\n[4] 3D Data Augmentation for Driving Scenes on Camera\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n- **Indoor/outdoor difference and novelty**:\nDifferent from outdoor scenes, the challenges in indoor scenes include (1) complex spatial layouts (in particular, cluttered background and limited object-placeable space) that necessitate a carefully-designed method to allow automated object placement (physically-plausible position, size, pose), and (2) complex lighting effects such as soft shadow, inter-reflection and long-range light source dependency that demand dealing with lighting for harmonious object insertion.\n\n  To deal with the above challenges, we make the following key technical contributions: (1) To avoid collision, after plane detection, we propose constrained insertion parameter search (algorithm 1) to guarantee a physically-plausible inserted position, pose and size. We also include an efficient collision check method to solve the time-consuming challenge. (2) We conduct environment map transformation and refinement to add more accurate illumination on inserted objects.\n\n  To our best knowledge, our work is the first automated 3D insertion pipeline on complex indoor scenes, enabling large-scale 3D insertion for data augmentation. It also showcases that physically plausible 3D object insertion can serve as an effective generative data augmentation technique for indoor scenes, leading to state-of-the-art performance in discriminative downstream tasks such as indoor monocular 3D object detection, opening up new avenues for research and practical applications.\n  \n  Thank you for your suggestion, we will modify the manuscript as follows: \n  - we already cite your ref [3] Neural Light Field Estimation but will add the other outdoor citations.\n  - we will tone down the claim that indoor scene is more challenging since it may indeed be hard to defend.\n  - we will extend our discussion of how our work compares to previous art with outdoor scenes. \n  \n  \n- **AP_0.25 and AP_0.15**:\nThis has to do with the fact that official results are only available for a few combinations of dataset setting, object classes, and IOU threshold: The official ImVoxelNet GitHub code, when using SUN RBG-D on “10 classes from VoteNet” setting (same as ours), also uses 0.25 IOU threshold with performance 40.7 (even though they use 0.15 with other datasets/classes). \nThe authors provided an implementation in MMdetection3D [1] Github, which also uses mAP 0.25 for the 10 classes from VoteNet setting (performance 40.96). We used the official code of MMdetection3D, so we used the 0.25 IOU threshold. Per your suggestion, we also show our results with mAP 0.15 on SUN RGB-D dataset (10 classes from VoteNet) below, our method shows consistent improvements.\n\n|SUN RGB-D        | mAP_0.15|\n|-----------------|------|\n| ImVoxelNet              |48.45 |\n| ImVoxelNet + 3D Copy-Paste|**51.16**|\n  \n  \n- **More experiments on outdoor dataset**:\nThanks! In this paper, we focus on indoor scene insertion, we will treat extension to the outdoor scene as a future work to explore. In the meantime, we note that our title and overall positioning of the paper make it very clear that it is currently focused on indoor scenes (i.e., we are not overselling the work).\n\n  However, we added experiments on the new dataset ScanNet[2] and new models (Implicit3DUnderstanding[3] in supplementary) as a way to show the generalization of our method. Here are the detailed setting and results on ScanNet:\n\n  [*Adapt to monocular setting*] ScanNet contains 1,201 videos(scene) in the training set and 312 videos(scene) in the validation set. For monocular 3D object detection, we use one RGB-D image per video, so 1,201 RGB-D images for training and 312 for validation (test). We calculate the ground truth 3D bounding box label for each of our used views from their provided video(scene) level label (because some objects in the scene may not be visible in our monocular view). \n\n  [*Training and test*] For the baseline, we train an ImVoxelNet monocular 3D object detection model on the training set and test on the validation set. For our method, there are 8 overlap categories (sofa, bookshelf, chair, table, bed, desk, toilet, bathtub) in the 18 classes of ScanNet with our collected Objaverse data (main paper Table1). We use our 3D copy-past to augment the training set and train an ImVoxelNet. All the training parameters are the same as the training on SUN RGB-D dataset. We will release all the code and training data.\n\n  We show the results on the average accuracy of the 8 overlap classes (AP_0.25) in the Table below. Our 3D Copy-Paste improves ImVoxelNet by 2.8% mAP.\n\n|ScanNet AP_0.25  | Average (mAP)|bed |chair|sofa|table|bookshelf|desk|bathtub|toilet|\n|---|--|--|--|----|----|----|----|----|----|\n| ImVoxelNet              |14.1 |25.7|7.9  |**13.2**|7.8  |4.2      |20.5|22.1   |**11.5**  |\n| ImVoxelNet+3D Copy-Paste|**16.9** |**27.7**|**12.7** |10.0|**10.8** |**9.2**      |**26.2**|**29.2**   |9.0   |\n\n\n- **Paper checklist**:\nThanks for your reminder! The limitations and broader impact discussion were in the supplementary, we also added more experiment details in the supplementary. We will add more experiment and implementation details and corresponding Licenses and Assets, and double-check the PaperChecklist.\n\n**References**\n\n[1] Contributors, M. (2020). MMDetection3D: OpenMMLab next-generation platform for general 3D object detection.\n\n[2] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T. and Nießner, M., 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5828-5839).\n\n[3] Zhang, C., Cui, Z., Zhang, Y., Zeng, B., Pollefeys, M. and Liu, S., 2021. Holistic 3d scene understanding from a single image with implicit representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8833-8842)."}, {"review_id": "D7HWIu357i", "reviewer": "Reviewer_mHBD", "summary": "This paper proposes a data augmentation approach to assisting the training of monocular 3D detectors by inserting 3D objects into indoor scenes in a physically plausible manner. Specifically, it addresses two main challenges in the entire pipeline: 1) where and how to insert those objects; 2) which illumination on the object makes the rendering photorealistic. Experiments validate that the data generated by the overall pipeline can enhance the state-of-the-art monocular 3D detectors. Detailed ablation studies further provide some insights regarding which aspects are important in the proposed method.", "questions": "None.", "limitations": "The author discusses the limitations and social impacts in the supplementary material.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The basic idea is easy to follow and the illustration figures are clear.\n- The overall pipeline is proven effective for enhancing downstream 3D perception systems.\n- This pipeline is systematic and comprehensive, taking almost most of the aspects when inserting 3D objects into scenes and image data generation for training monocular detectors.\n- The key insights regarding three challenges in the introduction and two critical considerations (where and how & illumination) in the methodology are accurate.\n- The proposed method achieves new state-of-the-art on the SUN RGB-D benchmark and has detailed ablation studies to reveal which aspects are most essential in the overall pipeline. (For example, the analysis in Table 4 is interesting.)", "weaknesses": "- (Related work) The related work section has many inaccurate statements, such as MV3D is a multi-view method incorporating both LiDAR-based point clouds and images and VoxelNet is a LiDAR-only method, which should not be discussed in the monocular 3D detection section. In contrast, there are many missing works of monocular 3D detection in driving scenarios, such as 3DOP[1], MLFusion[2], M3D-RPN[3], MonoDIS[4], Pseudo-LiDAR[5], FCOS3D[6], SMOKE[7], RTM3D[8], PGD[9], CaDDN[10], etc. There is also some missing literature for 3D Data Augmentation, such as MoCa[11], GeoAug[12], etc.\n\n- (Methodology) The overall pipeline is the main contribution of this paper. From another perspective, the most important part is to combine different \"existing\" techniques together and make it finally work to produce photorealistic images after inserting 3D objects. The key flaw here is that most of the techniques used in each stage are not newly proposed in this work, making this work more like an engineering one. (although I admit that the overall pipeline is still a good contribution to the community)\n\n- (Experiments) The proposed method is not limited to any method and any dataset, but only tested on ImVoxelNet and SUN RGB-D. Although it can demonstrate the basic effectiveness, it would be much more convincing if the author can provide more ablation results on other baselines and datasets.\n\n[1] 3D Object Proposals for Accurate Object Class Detection\n\n[2] Multi-Level Fusion Based 3D Object Detection from Monocular Images\n\n[3] M3D-RPN: Monocular 3D Region Proposal Network for Object Detection\n\n[4] Disentangling Monocular 3D Object Detection\n\n[5] Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving\n\n[6] FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection\n\n[7] SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation\n\n[8] RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving\n\n[9] Probabilistic and Geometric Depth: Detecting Objects in Perspective\n\n[10] Categorical Depth Distribution Network for Monocular 3D Object Detection\n\n[11] Exploring Data Augmentation for Multi-Modality 3D Object Detection\n\n[12] Exploring Geometric Consistency for Monocular 3D Object Detection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments! Please see our response below.\n\n- **Related work**:\nThanks for your suggestion and comments! We will modify the statement and add all the provided related papers.\n\n\n- **Methodology**:\nTo conduct 3D object insertion for data augmentation, traditional methods [1] require manually locating support planes, designing poses, estimating lighting, etc, which are hard to scale up. Out method is the first automated 3D insertion pipeline on complex indoor scenes, enabling large-scale 3D insertion for data augmentation.\n\n  To allow a fully automated pipeline, we make the following technical contributions: (1) To avoid collision, after plane detection, we propose a constrained insertion parameter search method (algorithm 1) to guarantee a physically-plausible inserted position, pose, and size. We also include an efficient collision check method to solve the time-consuming challenge. (2) We conduct environment map transformation and refinement to add more accurate illumination on inserted objects.\n\n  To our best knowledge, our work is the first to showcase that physically plausible 3D object insertion can serve as an effective generative data augmentation technique for indoor scenes, leading to state-of-the-art performance in discriminative downstream tasks such as monocular 3D object detection, opening up new avenues for research and practical applications.\n  \n- **More experiments on other models and datasets**:\n\n  **For more monocular 3D object detection methods**, we conducted experiments on Implicit3DUnderstanding (Im3D[2]) in supplementary materials, and the results are in supplementary Table S1. Using our method (Im3D + 3D Copy-Paste) improve the mAP_0.25 from 42.13 (Im3D) to 43.34.\n\n  **For more datasets**, we extend our method to ScanNet [3] dataset. Here are the detailed setting and results on ScanNet:\n\n  [*Adapt to monocular setting*] ScanNet contains 1,201 videos(scene) in the training set and 312 videos(scene) in the validation set. For monocular 3D object detection, we use one RGB-D image per video, so 1,201 RGB-D images for training and 312 for validation (test). We calculate the ground truth 3D bounding box label for each of our used views from their provided video(scene) level label (because some objects in the scene may not be visible in our monocular view). \n\n  [*Training and test*] For the baseline, we train an ImVoxelNet monocular 3D object detection model on the training set and test on the validation set. For our method, there are 8 overlap categories (sofa, bookshelf, chair, table, bed, desk, toilet, bathtub) in the 18 classes of ScanNet with our collected Objaverse data (main paper Table1). We use our 3D copy-past to augment the training set and train an ImVoxelNet. All the training parameters are the same as the training on SUN RGB-D dataset. We will release all the code.\n\n    We show the results on the average accuracy of the 8 overlap classes (AP_0.25) in the Table below. Our 3D Copy-Paste improves ImVoxelNet by 2.8% mAP.\n\n|ScanNet AP_0.25                   | Average (mAP)|bed |chair|sofa|table|bookshelf|desk|bathtub|toilet|\n|-----------------|------|------|----|----|----|----|----|----|----|\n| ImVoxelNet              |14.1 |25.7|7.9  |**13.2**|7.8  |4.2      |20.5|22.1   |**11.5**  |\n| ImVoxelNet+3D Copy-Paste|**16.9** |**27.7**|**12.7** |10.0|**10.8** |**9.2**      |**26.2**|**29.2**   |9.0   |\n\n\n\n\n**References**\n\n[1] Debevec, P., 2008. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Acm siggraph 2008 classes (pp. 1-10).\n\n[2] Zhang, C., Cui, Z., Zhang, Y., Zeng, B., Pollefeys, M. and Liu, S., 2021. Holistic 3d scene understanding from a single image with implicit representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8833-8842).\n\n[3] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T. and Nießner, M., 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5828-5839)."}]

Author rebuttal:
## Global response (common questions) to all reviewers

We would like to thank our reviewers, which put considerable time and thoughts for helping improve our paper. 
We are pleased that the reviewers find our paper "very well-written and easy to follow"(R-sM4b, R-mHBD, R-DWmW); our method being described as "novel" (R-DWmW), "fully automated" (R-B4nT), "impressive, well-thought-of and designed" (R-sM4b) and "systematic and comprehensive"(R-mHBD), with a "solid implementation"(R-7hAU); our experiments were commended as "extensive" (R-DWmW), "solid" (R-B4nT), backed by "supportive ablations" (R-7hAU, R-mHBD), demonstrating "effectiveness" (R-sM4b), achieving "state-of-the-art performance" (R-sM4b, R-mHBD), and notably improving "monocular 3D detection by a large margin" (R-DWmW).

Please find below our summary of major changes and responses to some common questions. We will incorporate these major changes into our revised paper.

**Summary of major changes**:
1. [7hAU, DWmW, mHBD] We add new experiments on the ScanNet[1] dataset to show that 3D Copy-Paste also improves monocular 3D object detection performance on other dataset. 
2. [7hAU, B4nT] We evaluate our monocular 3D object detection performance on SUN RGBD with mAP_0.15 and show consistent improvements.
3. [sM4b] We add detailed results of each individual object category on the SUN RGB-D dataset.


**Experiments on ScanNet dataset**:

To show the generalization to other datasets, we apply our 3D Copy-Paste to ScanNet[1] and conduct monocular 3D object detection with ImVoxelNet. ScanNet is a large-scale R-GBD video dataset that isn't specifically tailored for monocular 3D object detection. Here are the detailed experiments and results (we use ScanNet v2):

  [*Adapt to monocular setting*] ScanNet contains 1,201 videos (scenes) in the training set and 312 videos (scenes) in the validation set. Adapting it for monocular 3D object detection, we utilized one RGB-D image per video, amounting to 1,201 RGB-D images for training and 312 for validation. We calculate the ground truth 3D bounding box label for each of our used views from their provided video (scene) level label (because some objects in the scene may not be visible in our monocular view). 

  [*Training and test*] For the baseline, we train an ImVoxelNet monocular 3D object detection model on the training set and test on the validation set. For our method, there are 8 overlap categories (sofa, bookshelf, chair, table, bed, desk, toilet, bathtub) in the 18 classes of ScanNet with our collected Objaverse data (main paper Table1). We use our 3D copy-past to augment the training set and train an ImVoxelNet. All the training parameters are the same as the training on SUN RGB-D dataset. We will release all the code.

  We show the results on the average accuracy of the 8 overlap classes (AP_0.25) in the Table below. Our 3D Copy-Paste improves ImVoxelNet by 2.8% mAP.

|ScanNet AP_0.25                   | Average (mAP)|bed |chair|sofa|table|bookshelf|desk|bathtub|toilet|
|-----------------|------|------|----|----|----|----|----|----|----|
| ImVoxelNet              |14.1 |25.7|7.9  |**13.2**|7.8  |4.2      |20.5|22.1   |**11.5**  |
| ImVoxelNet+3D Copy-Paste|**16.9** |**27.7**|**12.7** |10.0|**10.8** |**9.2**      |**26.2**|**29.2**   |9.0   |


**Experiments on other 3D detection method**:

To show the generalization of our method to other downstream monocular 3D object detection methods, in our supplementary material Table S1, we conducted additional experiments with another monocular 3D object detection model: Implicit3DUnderstanding (Im3D [2]). Using our method (Im3D + 3D Copy-Paste) improve the mAP_0.25 from 42.13 (Im3D) to 43.34.

**Performance details of each category in SUN RGB-D dataset**:

In the table below, we show the detailed SUN RGB-D monocular 3D object detection results with ImVoxelNet of the main paper Table 2 on each individual object category:

|SUN RGB-D AP_0.25                   | Average (mAP)   |bed |chair|sofa|table|book shelf|desk|bathtub|toilet|dresser|night stand|
|-----------------|------|------|----|----|----|----|----|----|----|----|----|
| ImVoxelNet              |40.96 |72.0|55.6 |53.0|41.1 |**7.6**      |21.5|29.6   |76.7  |19.0|33.4|
| ImVoxelNet+3D Copy-Paste|**43.79** |**72.6**|**57.1** |**55.1**|**41.8** |7.1      |**24.1**|**40.2**   |**80.7**  |**22.3**|**36.9**|


**Reference**


[1] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T. and Nießner, M., 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5828-5839).

[2] Zhang, C., Cui, Z., Zhang, Y., Zeng, B., Pollefeys, M. and Liu, S., 2021. Holistic 3d scene understanding from a single image with implicit representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8833-8842).

Meta-review: All reviewers recommend acceptance. The AC sees no basis to overturn the reviews, and recommends acceptance. There was a discussion. Authors should attend to main points in the reviews, such as Reviewer 7hAU's questions about mAP (which was brought up by another reviewer during discussion), when preparing a final version.
Decision: Accept (poster)