Point-based Instance Completion with Scene Constraints
Authors: Wesley Khademi
Fuxin Li
Venue: ICLR 2025 Poster (2025)
Keywords: instance scene completion
point cloud completion
Primary area: learning on graphs and other geometries & topologies

Abstract: Recent point-based object completion methods have demonstrated the ability to accurately recover the missing geometry of partially observed objects. However, these approaches are not well-suited for completing objects within a scene, as they do not consider known scene constraints (e.g., other observed surfaces) in their completions and further expect the partial input to be in a canonical coordinate system, which does not hold for objects within scenes. While instance scene completion methods have been proposed for completing objects within a scene, they lag behind point-based object completion methods in terms of object completion quality and still do not consider known scene constraints during completion. To overcome these limitations, we propose a point cloud-based instance completion model that can robustly complete objects at arbitrary scales and pose in the scene. To enable reasoning at the scene level, we introduce a sparse set of scene constraints represented as point clouds and integrate them into our completion model via a cross-attention mechanism. To evaluate the instance scene completion task on indoor scenes, we further build a new dataset called ScanWCF, which contains labeled partial scans as well as aligned ground truth scene completions that are watertight and collision-free. Through several experiments, we demonstrate that our method achieves improved fidelity to partial scans, higher completion quality, and greater plausibility over existing state-of-the-art methods.

Published as a conference paper at ICLR 2025
POINT-BASED INSTANCE COMPLETION WITH SCENE CONSTRAINTS
Wesley Khademi & Li Fuxin
Oregon State University
(khademiw, fuxin. ( aoregonstate.
ABSTRACT
Recent point-based object completion methods have demonstrated the ability to accurately recover the missing geometry of partially observed objects. However, these approaches are not well-suited for completing objects within a scene, as they do consider known scene constraints (e.g., other observed surfaces) in their completions and further expect the partial input to be in a canonical coordinate system, which does not hold for objects within scenes. While instance scene com- pletion methods have been proposed for completing objects within scene, they lag behind point-based object completion methods terms of object completion quality and still do not consider known scene constraints during completion. To overcome these limitations, we propose a point cloud-based instance completion model that robustly complete objects arbitrary scales and pose in the scene. To enable reasoning at the scene level, we introduce a sparse set scene con- straints represented as point clouds and integrate them into completion model via a cross-attention mechanism. To evaluate the instance scene completion task indoor scenes, further build labeled partial scans as well as aligned ground truth scene completions that are watertight and collision-free. Through several experiments, demonstrate that our method achieves improved fidelity to partial scans, higher completion quality, and greater plausibility over existing nate-of-the-aad methods.
dataset called Scan which contains
RiD-Net
DIMR
Ours
Figure Visual comparison of completion results. Our approach better at recovering missing geometry, avoiding collisions, and preserving observed surfaces and known free space.
INTRODUCTION
Object modeling from sensor observations an increasingly prominent problem for robot interac- tion within real environments. However, sensors such as LiDAR and depth cameras provide with only partial observations of objects in scene. Even with dense scanning and reconstruction, many objects in reconstructed scene are left incomplete, parts of their geometry were completely oc- cluded during the scanning process. With lack of knowledge about the full 3D geometry objects, the performance of robot downstream tasks such navigation and grasp planning may be hin- dered because of limited capabilities to infer the center of gravity and other physical properties of objects.
Point-based 3D object completion has been researched for several years starting from Yuan et al. 2018). In recent years, the performance this task has improved significantly by leverag-

---

Published as a conference paper at ICLR 2025
ing encoder-decoder architectures, which first produce coarse completion, commonly known as seeds (Xiang et al.] 2021 Zhou et al. 2022) and then upsample it hierarchically. However, most of these work operate in canonically aligned coordinate systems i... setting the center of the com- plete object as having O coordinates and a scale of 1, and rotated so that a designated "front" side will always be along the direction. This is also applied the partial inputs that they match the normalization and pose of the completed object. These are unrealistic assumptions when objects are presented in scenes, but they indeed simplify the problem so that, in some sense, algorithms only have to match the partial object to complete objects in the training set, then directly generate the coordinates for the completion, as the canonical coordinates of the complete object will always be the same no matter which part is used as input.
On the other hand, semantic scene completion (Song et al. 2017 Liu et al. 2018 et al. 2019 Dong et al., 2023) has been studied heavily, but they only provide voxel labels without identifying objects. Hou et al. (2020) proposed the instance scene completion task, which involves detecting individual object instances in the scene and completing the missing regions of these objects. There
has been some work on this front (Nie et al. 1021 Tang et al. 2022 Li et al. 2023a), but the network structures have not been explored as much the point- based object completion task.
In this paper, we explore whether the more refined network structures of point-based object comple- tion models can be adapted to the instance scene completion task. We note two hurdles to this goal, the first is to lift the canonical coordinates assumption so that the completion model no longer has to know the scale and pose of the full object. The second is to make the instance completion to be aware of potential scene constraints that can be deduced from visibility. For example, if one sees surface, then the network should not add points on this viewing ray. Besides, if there is another object already present at some location, then the completion should not collide with those areas.
To solve these two difficult problems, we significantly improve the point-based completion frame- work mainly by adopting more sophisticated seed generator. First, we change the seed prediction to two parts, predicting an object center location and seed offsets from the center. We made several ar- chitectural improvements to improve performance in this setup to match the completion performance when using canonical coordinates. Next, to provide our completion model with scene context, we introduce a set of sparse constraints which encode known information about the scene. Unlike dense TSDFs, our constraints are represented as two bounding shells of the underlying surface, indicating the transition boundary from surface to known free space and surface to known occluded space. We integrate these constraints into our seed generator, enabling our model to reason about completions which are plausible in the context of the observed scene. Using this additional scene information, we find that our approach produces fewer collisions between predicted completions.
Furthermore, existing datasets for the instance scene completion task suffer from errors in the ground truth data, making evaluation on them unreliable. Scan2CAD (Avetisyan et al. (019a) lacks align- ment between real partial scans from ScanNet and synthetic ground truth meshes from ShapeNet, causing a trade-off between respecting the partial scan and matching the ground truth mesh. Scan- ARCW (Li al., 2023a) contains collisions their ground truth data, making it difficult to measure the plausibility of scene completion with collision metrics. We introduce new dataset called Scan- WCF addressing the issues present in these existing datasets. On our newly proposed dataset we demonstrate that our approach outperforms existing works in terms of both partial reconstruction quality and completion quality while producing less collisions between predictions.
In summary, our main contributions are follows:
We propose a novel object-level completion model which is robust to the scale and pose of objects found within scenes.
We integrate a sparse set of scene constraints into our model to provide our object comple- tions with scene context (e.g.. other observed surfaces, free space, occluded space).
We build a new dataset for instance scene completion on indoor scenes, containing partial scans and watertight ground truth meshes that are aligned, labeled, and collision free.
RELATED WORK
The 3D completion literature can be roughly broken down into two main categories: (I) object-level completion and (2) scene-level completion.
2

---

Published as a conference paper at ICLR 2025
Object-level Completion 3D object completion aims recovering the complete geometry of an object given some partial observation. With the introduction of point cloud architectures, Yuan et al. (0188) the first to propose fully point cloud based completion architecture. Since then, there
have been many follow-up works which leveraged increasingly better incoder-decoded architectures
(Tchapmi et al., 2019: [Liu et al. 2020 Wen al. 2020 Huang al. 2020 Wen al., [02T]] Yu
it al. 2021). y. In recent years, a popular choice has been to first produce a coarse completion, often
referred as seeds, and then upsample the completion in a hierarchical fashion (Xiang al. [021]
Zhou et al. 2022; Chen et al. [023] Khademi & Fuxin 2024). However, all these approaches oper- ate canonically aligned coordinate systems and thus are not suited for completing objects in the context of scenes where pose and scale are arbitrary. SCARP (Sen et al. 2003) is a point cloud com- pletion method that aims to robust to arbitrary pose by predicting the completion frame along with the pose needed to transform the completion back to its true coordinate system. Unlike SCARP, our method considers the scene context around the object as we are interested in the scene completion task. Additionally, our approach does not require estimating the pose of the object, avoiding possible misalignment with the partial scan due to inaccuracies in pose estimation.
canonical
Scene-level Completion Scene-level completion has heavily been studied joint task of seman- tically labeling the scene while recovering the geometric structures that are missing from it. For indoor environments, this task has been studied almost exclusively as predicting the semantic label and occupancy of each voxel in a dense voxel grid (Song et al. [017 Liu et al. 2018 Li et al.) 2019 Chen al., 2020 Cai et al., 2021; Dong al. 2023 Wang et al. 2024). These methods do not assign voxels to object instances in their completion, and use dense 3D convolutions to produce predictions on low-resolution voxel grids, prohibiting the representation of fine-grained geometry
typically present in indoor environments.
Avoiding the need for dense occupancy and semantic prediction, some methods produce comple- tions of partial scans by performing CAD model retrieval and alignment (Avetisyan et al.] 2019b] Ishimtsev et al. 2020). However, these approaches require the existence CAD pool, have to search for the nearest CAD model for each object in the partial scan, and potentially require an additional optimization for roughly aligning the models to the scan.
To improve the level of scene understanding required for object interaction within environments, Hou et al. (2020) proposed the instance scene completion task, which involves detecting individual object instances in the scene and completing the missing regions of these objects. Their approach relies on producing occupancy predictions and semantic labels for a dense voxel grid using 3D convolutions, limiting the resolution of scenes due to memory footprint. More recent instance scene completion methods operate directly on point clouds. RfD-Net (Nie et al. 2021) generates instance proposals via 3D object detection and completions via implicit function. DIMR (Tang et al., 2022) trains a 3D instance segmentation model for proposal generation and produces completions from latent codes with a pre-trained shape generator. DDIT (Li et al. 2023a performs 3D instance segmentation and deforms deep implicit shape templates into completions conditioned on shape latent codes extracted from the segmented partial objects. However, DDIT requires an iterative procedure for estimating object pose and per-scene optimization step for respecting the partial input which slow. PaSCo (Cao et al., (024) performs panoptic scene completion on LiDAR scans of outdoor scenes. However, they employ a dense 3D CNN in part of their network, making their work infcasible for our indoor scenes due to memory constraints (our 2cm resolution indoor scans would require - 5x more voxels than the outdoor scans with 20cm voxels used in their work).
3 METHOD
We present an overview of our architecture in Figure 2 Given a partial scan of a scene, we run a state-of-the-art 3D instance segmentation method Mask3D (Schult al. 2023) to decompose our scene into a set of objects. The partial information of each object is first encoded into set of local features and a global shape descriptor. We then introduce a novel seed generator which generates coarse representation of the complete shape, called Patch Seeds, from our partial encoding. Our generator produces Patch Seeds as offsets of the object's predicted center, which we find to be more robust to change in pose compared to existing Patch Seed generators. To provide object completions with scene context, we additionally integrate a set of scene constraints into our seed generator via cross-attention. Generated Patch Seeds are then decoded into dense completion in
3

---

Published as a conference paper at ICLR 2025
Stage Instainee Segmentation
Stage Onject Completion
Stage Reconstruction
Fartial
Eired
Sand
Frea Space Polrts
Oifer
Dockuded Space Points
Concrnrte
Figure 2: Overview of our instance scene completion framework. Instance segmentation is first performed on the partial sean to decompose the scene into its individual objects. Each object is run through our proposed object completion model, which predicts both the complete shape and surface normals. Meshes each object are then reconstructed to produce the completed scene.
hierarchical fashion by applying a series of upsampling layers. We design upsampling layer to contain both local and global attention, which helps with producing a globally coherent completion capable of representing fine-grained geometry. At the densest completion level, we additionally leverage a local transformer to predict surface normals, allowing us to reconstruct meshes of our completions using an off-the-shelf surface reconstruction method. Finally, we place each mesh back into the world frame, producing scene of labeled object instances with complete geometry. In the following sections, we describe the key components of our completion model in more detail.
3.1 PARTIAL ENCODER
Our partial encoder takes as input the partial object instance normals N E RMx3 to produce downsampled set of points p E MR with local features F. E X.MMxCrnnaa and a global shape descriptor fp E IR Cotobat from the object. We base our encoder on the design proposed by Khademi & Fuxin (2024), which is in turn based on Zhou et al. (2022). The encoder consists of downsampling blocks, where at each block the point set is first downsampled, followed by series of point convolutions for interpolating and aggregating features for the downsampled point set. The final downsampled set of features are then passed through an ML.P followed by max-pooling operation to produce global shape descriptor fp-
RMx and estimated surface
The main improvement we made to the partial encoder is that we replace the PointConv (Wu et al., 2019) layers in Khademi Fuxin (2024) with VI-PointCony eLi al. 2023b) layers and addition- ally use estimated normals as input. The convolution filters generated in PointConv are only trans- lation invariant. To be robust to rotation and scale, the network would have be shown the same object under many different rotations and scales and have enough capacity to encode appropriate filter weights for each of them. the other hand, the convolution filters in VI-PointConv are gen- erated from mix of non-invariant, scale-invariant, and votation-invarian position embeddings. In this way, the MI.P generating filter weights can potentially learn ignore the non-invariant position embeddings to share filter weights across neighborhoods of different scales and rotations, increasing robustness of the network. Furthermore, using both XYZ coordinates and surface normals provide better description of local surface geometry than XYZ coordinates alone.
3.2 SEED GENER ATOR
Given the downsampled point set Pt, locally extracted features F and global shape descriptor fp- our seed generator is tasked with producing set of Patch Seed coordinates E IR x3 and features Freed E IR M which represent coarse encoding of the complete shape.
SeedFormer (Zhou al. 2022) produces Patch Seeds through a local attention based upsampler fol- lowed by MIP to regress the seed coordinates. In Table we show that such an approach suffers a significant drop in completion quality if we attempt to use complete objects not canonically aligned along axis. Instead, design global attention-based seed generator, which allows the generator to use information from the entire scene to predict the center the object before producing seed coordinates as offsets of the object center. Our full architecture is shown in Figure 3
We introduce a learnable token Otoken E RC which is to be decoded into the center of an object O E R Our learnable token Otoken along with downsampled partial information {P' F is first passed through set of transformer blocks performing multi-head self-attention to produce an updated object center embedding Ooby E CRC and updated partial features Fatj R MC Having
4

---

Published as a conference paper at ICLR 2025
Scenn Thansformer Block (+?)
yrer
R
Cross anition
Seed Offict Prediction
Transposed Conv
Self Aftention
HLP
Offset
010
Oacene
Offacia
Object Block (x4)
Ooject
Pati
4
Praltional Encading
Summation
f
Concetioation
Figure 3: Overview of our proposed seed generator. Predicting Patch Seed coordinates offsets from the shape's predicted object center is more robust than directly regressing seed coordinates as in Zhou et al. 20022). Our object completions additionally consider other objects in the scene through cross-attention with our known free and occluded space constraints.
aggregated information from the partial input via attention, Oobj is then used to regress the object center. We concatenate Oobj with the global shape descriptor fp along the feature dimension before predicting the object center O with an MLP 0:
O 0 {(obbj fpl)
To reliably cover the entire object in our coarse representation, increase the number of points present in our Patch Seeds compared to the partial points and features from which they are generated. Specifically, we "split" our output partial features Fobj using transposed convolution with a stride and kernel size of 2. Upsampled features are then passed through a multi-head self-attention block followed by an MLP w to produce Patch Seed features Fseed:
Fseed = w =TSSeffttttTTaanppssCCCoo//
Finally, Patch Seed coordinates S are predicted as offsets from object center O. y which predicts per point offsets from Patch Seed features Fseed concatenated with object center token Jobj and global shape descriptor fp along the feature dimension:
We use a MLP
S = O + r([Fseed, Oobj. fpl)
3.3 SCENE-AWARK OBJECT COMPLETION
As mentioned earlier, correctly completing objects a scene requires satisfying visibility constraints from the scene, i.c. not creating object parts that cut into other objects or grow towards the camera (violating the free space that has been seen). TSDFs are commonly used for representing known information in scene, such as the surfaces of objects well as known free occluded space due to the viewpoints at which the scene was captured. However, TSDFs are typically stored as dense voxel grids, making them computationally expensive to process and not compatible with point clouds. To our knowledge, no one has attempted to address the difficult task of incorporating scene constraints to posed completion. We propose to represent the constraint information in the scene sparse sets of points representing known occupied and free spaces. Our goal is to input these constraint points the network so that it can learn to avoid generating parts in those areas.
We generate our scene constraints as two bounding shells the surface of the partial scan defined P.,. + 5N,.. where P.,. R Mi. is the partial scan with surface normals N E M., x3 and 8 E R. We further resample these surfaces to 10 cm resolution, leaving us with a sparse set of free space points Pfree E pr X3 and occluded space points Poee E Ad Along with these point sets, learn additional embeddings e free C and eoce CRC which shared across all free space points and occluded space points, respectively.
We provide our constraints as information our seed generator through a set of transformer blocks containing multi-head cross-attention as shown in Figure 3 Here the object's partial information is treated as the query tokens while the scene constraints represent the key-value pairs, allowing each partial object to decide what constraints to focus on when completing the object. The transformer blocks output set of features F Fscene E R M,X' and Osoene E CRC, which are directly added to the outputs Fatj and Ooby. respectively, before being used to predict the object's 's center and Patch Seeds.
5

---

Published as a conference paper at ICLR 2025
Scan2CAD
ScanARCW
ScanWCF (ours)
Figure 4: Our proposed Scan WCF has aligned ground truth meshes and partials scans while being free of collisions, unlike previous datasets Scan2CAD and ScanARCW.
3.4 COARSE-TO TRASSE-TO-FIIE FINE DECODER
For upsampling, we use the upsampling layer proposed by Zhou et al. (2022). In each upsampling layer, we further introduce series of global attention layers before their local attention-based Up- sample Transformer. With our added global attention, we provide the refinement and upsampling layers with information which can encourage global coherence across the completion. Additionally, global attention allows to potentially learn fine-graines structure for the missing geometry from the existing geometry which may not be present in the local neighborhood (c.g., through symmetric part the opposite side of the object).
Starting from our Patch Seed coordinates S and features F Fseed, we repeatedly apply our upsampling layer to generate dense completion in a coarse-to to---to---eeee- manner. At each upsampling layer j. produce a completion C and the corresponding upsampled features F with double the resolution of the previous layer. We apply 3 upsampling layers upsample our completion from coarse resolution of Mcoaree = 256 points to Mdense = 2, 048 points. The completion produced at each layer is supervised by the ground truth completion C₂ which has been subsampled to the same resolution the output resolution layer (we additionally treat the Patch Seed coordinates as our coarsest resolution C during training).
3.5 MESH RECONSTRUCTION
We used NKSR (Huang al. 2023 reconstruct meshes from point clouds. NKSR requires both point clouds and surface normals, where normals are typically estimated from the point cloud us- ing PCA-based plane fitting. practice, we found these normals poor reconstruction quality on completions. To address this, introduce normal estimation module which is jointly trained with our completion network. We first process our final completion C E Mannse x3 and corresponding upsampled features F I Maense xC using modified ver- sion of the Upsample Transformer from SeedFormer (Zhou et al.) (002)), replacing the transposed convolution used for upsampling with regular convolution. The features output from this layer encode local surface information of the object and are directly used as input to a small MLP that regresses surface normals 3 E IR Maneee
overly noisy, leading to
3.6 Loss
We use the same loss function for both pre-training the object completion model and training our scene completion model. Our overall loss objective is defined as:
Ae
CD CD' C.)) + Ap Σ OOD OOD (P,C') ) MSE (0, Oge) + An C (NN3 Ng)
(4)
4=00
where LCD is the Chamfer Distance between predicted and ground truth completions at each up- sample layer j. LOCD the One-sided Chamfer Distance between partial input and completion at each upsample layer j, MSEE is the mean squared error between predicted object center and ground truth object center O, E IR3, and LCS cosine similarity based loss between the predicted normals 3 and ground truth normals N experiments.
gr- We set X₂ = 1, Ap = 1, Ao 1, A, = 10 for

---

Published as a conference paper at ICLR 2025
4 SCANWCF DATASET
Previous works evaluate the instance scene completion task on the Scan2CAD dataset (Avetisyan) et al. 2019a). The Scan2CAD dataset is derived from both the ScanNet (Dai al. 2017) and ShapeNet (Chang et al. 2013) datasets, where for each detected partial object in a Scan scene, the closest synthetic mesh from ShapeNet has been selected and fit the scan to serve as the ground truth completion. The synthetic meshes are not the true completion of the partial object and their fit to the partial sean imperfect as shown in Figure The lack of alignment between the input and ground truth makes the evaluation of metrics unreliable on this data. To address this, [L et al. (203a)) proposed SeanARCW which regenerates new partial scans by rendering depth maps and semantic labels of the ground truth meshes and then backprojects this information back into 3D to generate an aligned scan. While ScanARCW addresses the alignment issue, the ground truth scenes in their dataset contain collisions. This makes it unreliable to measure scene completion plausibility with collision metrics, as the collision may be an artifact of the ground truth data itself.
To address the limitations of existing datasets, we introduce new dataset called ScanWCF, where "WCF" refers to our ground truth scenes being" "Watertight and Collision Free". Our dataset contains scenes with partial scans and ground truth complete meshes that are aligned and labeled, with each ground truth scene being watertight and collision free. We generate our scenes from data included in both the Scan2CAD and ScanARCW datasets. We use the background meshes from Sca as scene boundaries (e.g.. walls, floor, ceiling), filling holes in the mesh to make it completely enclosed. For selecting which objects to place in the scene, we use the Scan2CAD object matchings. Each ground truth mesh is processed to be watertight and initialized in the scene using the pose and scale from Scan2CAD. We then optimize the pose and scale of each object the scene such that they are: (1) closely aligned with the partial scan from ScanNet, (2) do not contain any collisions, and (3) are not floating in air. After optimization, we manually verify the scene is collision-fre free before includ- ing it in our dataset. If there exist minor implausibilities after optimization, scene; otherwise we completely discard it if too many collisions exist. After our optimization and manual verification, we find that average only 0.14% of the points in the scene are in collision with another object, compared to an average of 2.5% of points per scene in the ScanARCWW dataset. Finally, to generate aligned and labeled partial scans, we render depth maps and instance segmenta- tion maps from a subset of camera poses in the ScanNet camera trajectory. This information is then backprojected into 3D to generate partial scan labeled with instance information.
manually correct the
Our dataset contains 1202 indoor scenes based on ScanNet scenes, where 946 of the scenes are used for training and the other 246 scenes are reserved for testing. To increase the amount of training and test data, we generate 2 partial scans per scene using a different subset of camera poses from the ScanNet camera trajectory. We refer readers to our appendix for more details.
EXPERIMENTS
In this section, we perform a variety of experiments to demonstrate the superiority of our method over existing approaches. We first evaluate our method on the task of instance scene completion, where the goal is to jointly predict object instances and their completions. Secondly, we evaluate completion quality in isolation by removing the instance prediction task and instead providing the ground truth instance information directly to each method. Across both tasks, we show that our method outperforms existing approaches both quantitatively and qualitatively. Finally, conduct a set of ablations which validate the design choices of our architecture.
5.1 IMPLEMENTATION DETAILS
Our object-level completion model is pre-trained on the 34 ShapeNet categories present in our dataset, excluding objects present only in the validation scenes. For augmentations, we perform random rotations about the up-axis. For optimization, we use Adam with an initial learning rate of 1 xx 1 and linearly decay by factor of 0.98 every epochs. We train for 150 epochs with a batch size of 64 using 2 NVIDIA V100 GPUs, which takes approximately days. Our scene completion model is then trained on the ScanWCF dataset, initialized from the weights of our pre-trained object completion model. For training, we use the same augmentation and optimization setup as used for pre-training. We train for 200 epochs single RTX 4090 GPU which takes about 3 days.
7

---

Published as a conference paper at ICLR 2025
Table 1: Instance scene completion quality, Mean average precision (mAP) is reported for differ- ent metric@threshole
1oUR0.
CDB0.1
CD#000047
1FDe
LFFD
PCR
PCR00077
RED Net
38.78
1.25
61.68
31.32
39.18
14.65
59.56
42,36
DIMR
41.15
23.30
27.14
31.14 43.37
pre tranning)
Ours (w) pee triningg
62.77
47.55
51.57
36.07
51.29
29.03
63.71
62.57
5.2 BASELINES
We evaluate our proposed approach against instance scene completion methods RfD-Net (Nie al, 2021] and DIMR TTng et al., 2022). Both methods are trained on our ScanWCF dataset using the same hyperparameters that were used for their original experiments on the Scan2CAD dataset. We additionally retrain the pre-trained shape generator used in DIMR, but on the 34 categories from ShapeNet present in our dataset rather than the ShapeNet categorics it was originally trained on.
5.3 METRICS
For the instance scene completion task, we follow Tang al. (0022) and compute the mean average precision (mAP) across several different metrics and at varying thresholds. Specifically, Intersection over Union (IoU), Chamfer Distance (CD), and Light Field Distance (LFD) are used for evaluating completion quality while the Point Coverage Ratio (PCR) metric is used for evaluating partial re- construction quality. More details of those metrics are presented in the appendix. For IoU and PCR, higher thresholds are more challenging, while for CD and LFD lower thresholds more difficult.
To remove the possibility that poor completions are caused by worse instance segmentation, we evaluate completion quality when each method uses the ground truth instance masks. For par- tial reconstruction quality, we compute Unidirectional Hausdorff Distance (UHD) and One-Sided Chamfer Distance between the partial scan and the predicted complete scene. To evaluate comple- tion quality, we use the Chamfer Distance (CD) between points uniformly sampled from the scene mesh and the predicted completion. Finally, evaluate the plausibility of scene completions by measuring collisions between predicted completions. For each object, we sample points from its completion and compute their signed distance all the other predicted objects in the scene and the scene background mesh, penalizing points which fall inside another object mesh or outside the background mesh. We measure both the average distance of collisions (COL) and the percentage of points in collision (%COL). We scale UHD, One-Sided CD, and CD by 10 and scale COL by 104.
5.4 RESULTS
Instance Scene Completion We present results on the instance scene completion task in Table Despite the fact that do not jointly train instance segmentation network together with our completion model, unlike RiD-Net and DIMR, we outperform both approaches in terms of mAP across almost all metrics and thresholds. Both RfD>Nett and DIMR suffer large drop-offs in performance for all metrics in the transition from easier threshold to more difficult threshold. This suggests that these approaches can produce completions that share some similarities the ground truth shape, but likely cannot represent fine geometric details while also suffering from inaccuracies in their prediction of object's pose and scale. On the other hand, see much smaller drop- offs in our method for both CD and PCR, suggesting better ability to represent the shape's surface accurately. Furthermore, the pre-trane pre-training of our object completion model is not strict requirement,
I
Partial Scan
RiD-Net
DIMR
Ours
Ground Truth
Figure 5: Qualitative comparison on the instance completion task.
8

---

Published as a conference paper at ICLR 2025
Scene completion quality. Evaluation of scene completion quality metries when ground truth instance masks are used for all methods.
Table 2:
UHD
One-sided CD
CD
COI.
% &.COL.
RID Net
190.65
8.58
DIMR
270.67
44,91
39.04
10.42
4.63
Ours (no pro-training)
60.56
12.50
21.22
3.42
1.74
Ours (w/ pre-training)
60,07
12.10
20.74
2.67
1.81
we outperform both RID-Net and DIMR without it; however, do find that pre-training helps improve completion quality at the more challenging metric thresholds.
In Figure 5 we share an example of instance scene completions to represent thin geometric structures such as the base of a rolling chair, merging the wheel bases into almost solid circular base. DIMR suffers from low-fidelity completions, with many objects appearing to be a composition of planar primitives and failing to produce thin structures such as the legs of the chairs. On the other hand, our approach can faithfully represent the fine- fine-grained geometry present in the partial scans while producing plausible hallucinations of the missing regions (c.g., wheel bases of chairs). We share more qualitative results Figures and of our appendix.
partial scan. RID-Net struggles
Scene Completion To isolate our evaluation of completion quality, we present results on scene completion when the ground truth instance information has been provided to each method. In this setting, poor completion quality cannot be attributed to incorrect instance predictions, enabling us to better understand the limitations of the completion network of each method. In Table [
CD and UHD measure the predicted completions average and maximal deviation from the surface of a partial scan, respectively. Both of these metrics indicate that RfD-Net and DIMR are significantly worse respecting the partial input in comparison to approach. Similarly, we find that method is capable of producing higher quality completions than previous approaches, reflected by the significant gap in CD. Finally, method does much better job at avoiding collisions between predictions. In particular, RíD-Net and DIMR tend to produce completions that penetrate much further into other objects in the scene, producing an average collision distance (COL) which is 3 4- larger than our method. This is further demonstrated by RfD-Net and DIMR having 2 - 3% more points from predicted completions either in collision extending outside the scene boundaries compared to our approach. In Figures | and 6 we qualitatively demonstrate that our approach achieves higher fidelity to the partial scan, better completion quality, and produces less collisions than previous methods. We refer readers to our appendix for more qualitative results.
5.5 ABLATION STUDIES
In Table we conduct an ablation of our object completion model on the chair category from ShapeNet to justify our proposed additions for completing objects under arbitrary pose and scale. Our baseline method is SeedFormer (Zhou al.] [022] with their partial encoder replaced by the PointConv encoder proposed in (Khademi & Fuxin 2024). This replacement suffers no drop in per- formance and allows us to later update the PointConv layers for VI-PointConv layers our work. When trained and tested under ideal conditions (i.e., normalizing the input by the ground truth com- plete shape and having the object canonically aligned along a shared axis) our bascline is capable of producing high quality completions. When our baseline is trained and evaluated on more realistic setting (i.e., normalizing by the partial input and objects containing arbitrary pose and scale) we see that performance drops in both completion and partial reconstruction quality. We find that adding VI-PointConv (Li et al.) 2023b) to our partial encoder improves partial reconstruction quality, as the added rotation and scale invariant features likely enable better reasoning about the partial input.
RiD-Net
DIMR
Ours
Figure 6: Results on scene completion when ground truth instance information is provided.

---

Published as a conference paper at ICLR 2025
Table 3: Ablation of our proposed object completion network. Each row containing + additionally contains all the modules in the rows above it.
One-sided CD
CD (Scedk)
CD (Dense)
Baselime using canonical coordinates
41.69
Baseline
7.12
47.00
22,87
VI-Point Point & Input Normals
6.04
46.47
22.57
Object Center & Seed Offset Prediction
6.28
42.86
21.11
Global Attention Upsample Layers
6.18
43.71
20.05
Surface Normal Prediction
6.19
43.26
20.08
Table 4: Ablation of our scene completion network with and without scene constraints.
UHD
One-sided CD
CD
CO1
& COL.L
wña comstraints
60.18
12.20 12,10
1.83 22.37
3.75
w/ constraints
60.07
20.74
2.67
1.81
W/out constraints
W/ constraints
W/out constraints
W/ constraints
Figure 7: Comparison of our completion model with and without considering scene constraints.
Additionally, with the introduction of global attention in the seed generator and producing seed co- ordinates by first regressing the object center followed by predicting seeds as offsets from the center, we see the coarse and dense completion quality significantly improve over the old seed generator. Furthermore, incorporating global attention into the upsampling layers allows us to not only match our baseline's ' performance under ideal conditions, but actually beat it even though we are under non-ideal conditions. Finally, find that our surface normal prediction module neither harms helps our completions; however, it provides us with a way to reconstruct object meshes.
In Table evaluate the importance of leveraging scene constraints. We observe that using scene constraints does not improve partial reconstruction quality (UHD and One-Sided CD) as the comple- tion model does not need information about free space and occluded space reconstruct the already observed portion of the object. However, we find that incorporating scene constraints into our model produces 7% relative improvement in completion quality (CD) and 29% relative improvement in how far points in collision are penetrating into each other (COL). The scene constraints provide our model with information about the scene boundaries as well other objects in the scene, which helps better constrain our completions to ones which are plausible within the scene as shown in Figure 7
CONCLUSION
We present novel scene completion framework which obtains state-of-the-ar performance on the instance scene completion task for indoor scenes. Our proposed object completion model is robust to arbitrary pose and scale, enabling our method produce high-quality completions with high fidelity to the partial input without having to rely on accurately estimated pose and scale parameters needed transform the object canonically aligned axis. Furthermore, our proposed scene constraints enable our completion model to incorporate scene context in our completions, improving completion quality and reducing collisions between predicted completions. To evaluate our approach, we build new dataset for the instance scene completion task indoor scenes called Scan CCF, which contains collision-fii free scenes whose partial scans and ground truth meshes are aligned and labeled. Through several experiments, demonstrate method achieves higher completion quality, greater fidelity to the partial scan, and better plausibility over existing approaches.
Our completion framework is deterministie, meaning that approach only produce com- pletion of partial scan. In the future, we plan to explore incorporating generative models into our completion framework in order to be able to produce multiple plausible completions of a scene.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work partially supported by NSF under grants 1751402, 2321851, ONR under grant N0014- 21-1-2052, and DARPA under grant HR001 12490423.
REFERENCES
Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X Chang, and Matthias Nießner. Scan2ead: Learning cad model alignment in rgb-d scans. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 2614-2623, 2019a.
Armen Avetisyan, Angela Dai, and Matthias Nießner. End-to-end cad model retrieval and 9dof alignment in 3d scans. In Proceedings of the IEEF/CVF International Conference computer vision, pp. 2551 2019b.
Yingjie Cai, Xuesong Chen, Chao Zhang, Kwan-Yee Lin, Xiaogang Wang, and Hongsheng Li. Semantic scene completion via integrating instances and scene in-the-loop. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 324-333, 2021.
Anh-Quan Cao, Angela Dai, and Raoul de Charette. Pasco: Urban 3d panoptic scene completion with uncertainty awareness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14554-14564, 2024.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information- 3d model repository. arXiv preprint arXiv: 2015.
Ding- Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retrieval. Computer graphics forum, volume 22, pp. 223-232. Wiley Online Library, 2003.
Xiaokang Chen, Kwan-Yee Lin, Chen Qian, Gang Zeng, and Hongsheng Li. 3d sketch- se- mantic scene completion via semi-supervised structure prior. Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 4193-4200 2020.
Zhikai Chen, Fuchen Long, Zhaofan Qiu, Ting Yao, Wengang Zhou, Jiebo Luo, and Tao Mei. An- chorformer: Point cloud completion from discriminative nodes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1358 13581-13590, 2023.
Angela Dai, Angel X Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias
Nießner. Scannet: Kinhly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Haotian Dong, Enhui Ma, Lubo Wang, Miaohui Wang, Wuyuan Xie, Qing Guo, Ping Li, Lingyu Liang, Kairui Yang, and Di Lin. Cvsformer: Cross-view synthesis transformer for semantic scene completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8874-8883, 2023.
Ji Hou, Angela Dai, and Matthias Nießner. Revealnet: Seeing behind objects in rgb-d scans. In Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 2098- 2107, 2020.
Jiahui Huang, Zan Gojeic, Matan Atzmon, Or Litany, Sanja Fidler, and Francis Williams. Neural kernel surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1369-43799 2023.
Zitian Huang, Yikuan Yu, Jiawen Xu, Feng Ni, and Xinyi Le.. Pf-net: Point fractal network for 3d point cloud completion. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 7662-7670, 2020.
Vladislav Ishimtsev, Alexey Bokhovkin, Alexey Artemov, Savva Ignatyev, Matthias Niessner, Denis Zorin, and Evgeny Burnaev. Cad-deform: Deformable fitting of cad models to 3d scans. In Computer Aision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp. 399-628. Springer, 2020.

---

Published as a conference paper at ICLR 2025
Wesley Khademi and Li Fuxin. Diverse shape completion via style modulated generative adversarial networks. Advances in Neural Information Processing Systems, 36, 2024.
Haoang Li, Jinhu Dong, Binghui Wen, Ming Gao, Tianyu Huang. Yun-Hui Liu, and Daniel Cremers. Ddit: Semantic scene completion via deformable deep implicit templates. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21894-21904, 2023a.
Jie Li, Yu Liu, Dong Gong, Qinfeng Shi, Xia Yuan, Chunxia Zhao, and Ian Reid. Rghd based dimensional decomposition residual network for 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7693-7702, 2019.
Xingyi Li, Wenxuan Wu, Xiaoli Z Fern, and Li Fuxin. Improving the robustness of point convo- lution on k-nearest neighbor neighborhoods with a viewpoint-invariant coordinate transform. In Proce Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1287- 1297, 2023b.
Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu. Morphing and sampling network for dense point cloud completion. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 11696, 2020.
Shice Liu, Yu Hu, Yiming Zeng. Qiankun Tang, Beibei Jin, Yinhe Han, and Xiaowei Li. See and think: Disentangling semantic scene completion. Advances in Neural Information Processing Systems, 31, 2018.
Yinyu Nie, Ji Hou, Xiaoguang Han, and Matthias NieBner. Rfd-net: Point scene understanding by semantic instance reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4608-4618, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv:2007.0850/ 2020.
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. 2023.
Bipasha Sen, Aditya Agarwal, Gaurav Singh, B Brojeshwar, Srinath Sridhar, and Madhava Krishna. Scarp: 3d shape completion in arbitrary poses for improved grasping. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 3838-3845. IEEE, 2023.
Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In Proceedingy of the IEEE conference on computer vision and patlern recognition, pp. 1746-1754, 2017.
Jiaxiang Tang, Xiaokang Chen, Jingbo Wang, and Gang Zeng. Point scene understanding via disen- tangled instance mesh reconstruction. In European Conference on Computer Vision, pp. 684-701. Springer, 2022.
Lyne P Tchapmi, Vineet Kosaraju, Hamid Rezatofighi, Ian Reid, and Silvio Savarese. Topnet: Structural point cloud decoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 383- 383-392, 2019.
Fengyun Wang, Qianru Sun, Dong Zhang, and Jinhui Tang. Unleashing network potentials for semantic scene completion. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 10314-10323, 2024.
Peng-Shuai Wang. Yang Liu, and Xin Tong. Dual octree graph networks for learning adaptive volumetric shape representations. ACM Transactions Graphics (TOG), 41(4):1-15, 2022.
Xin Wen, Tianyang Li, Zhizhong Han, and Yu-Shen Liu. Point cloud completion by skip-attention network with hierarchical folding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1939-1948, 2020.
12

---

Published as a conference paper at ICLR 2025
Xin Wen, Peng Xiang, Zhizhong Han, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Yu-Shen Liu. Pmp-net: Point cloud completion by learning multi-step point moving paths. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7443-7452, 2021.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 9621-9630, 2019.
Peng Xiang, Xin Wen, Yu-She (u-Shen Liu, Yan-Pei Cao. Pengfei Wan, Wen Zheng, and Zhizhong Han. Snowflakenet: Point cloud completion by snowflake point deconvolution with kip-transformer. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 5499-5509, 2021.
Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 12498-12507, 2021.
Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. 2018 international conference on 3D) vision (3DV), pp. 728-737. IEEE, 2018.
Haoran Zhou, Yun Cao, Wenqing Chu, Junwei Zhu, Tong Lu, Ying Tai, and Chengjie Wang. Seed- former: Patch seeds based point cloud completion with upsample transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceed- ings, Part III, pp. 432..42.6622 Springer, 2022.
A
APPENDIX
An overview of our appendix is presented follows:
Model details (Section B we provide more details of our scene completion model
ScanWCF dataset (Section C we provide a detailed description of our proposed dataset
Metrics (Section D): we formally define our evaluation metrics
Results (Section share more results and ablations of our method
B MODEL DETAILS
B.1 PARTIAL ENCODER
Our partial encoder consists of = downsampling blocks, where each block contains down- sampling operation on the point set followed by 2 VI-PointCony (Li et al. 20235) layers. For point convolutions, use neighborhood size of 16. We design encoder such that the final downsam- pled point set pa E M334 and corresponding local features F E Ma have M4 = 128 points and local feature dimension size Ciocal 256. From downsampled points p4 and local features F.. we extract global shape descriptor fp E Cytobul through 2-layer MIP followed by max-pooling. We set the global feature dimension size to be Cglobal = 512.
B.2 SEED GENERATOR
Our seed generator produces Patch Seeds with coordinates S E R.Mared and features Fseed E R. We define our Patch Seeds to have double the resolution of the downsampled partial information they are generated from (i.c., Mased = 256) and set the seed feature dimension size to be Creed 256. All of the multi-head attention layers in seed generator hyperparameters, consisting of 8 heads where each head performs attention on features of dimension size 48. We define our learned embeddings Otoken CCC efere E C.C. and Cace RCC have an embedding size of C 384. We positionally encode downsampled partial points P4, free space points Prree. and occluded space points Poce via a 2-layer MLP, mapping the points to features of the same dimension size MLP, and the seed offsets which are added to the object center are predicted via a separate 3-layer ML.P.
the same
learned embeddings. The object's 's center is predicted by 3-layer
13

---

Published as a conference paper at ICLR 2025
B.3 COARSE-TO-FINE DECODER
Our decoder uses the upsampling layer proposed by SeedFormer Zhou et al. (2022) for producing dense completion of object in a hierarchical fashion. We neighborhood size of 20 when computing local attention in the Upsample Transformer of their upsampling layer. For our added global attention layers, we once again use multi-head self-attention with 8 heads. We upsample our completion 3 times, doubling the number of points present in the completion at each upsampling layer, to produce dense completion of Mdense 2048 points.
B.4 SCENE CONSTRAINTS
Scene constraints are generated from the input partial scan Psn. IR. x3 and its estimated surface normals Nin E R M., x3, As described in Section C.4 our partial scans are produced by backpro- jecting depth maps from 10 different viewpoints and resampling the scene to a cm resolution. As we are backprojecting a depth map into a point cloud, we also estimate surface normals, orienting them towards the camera the point cloud generated from that the normals are pointing wards "free space". With our partial points and estimated normals, we generate constraint points as P... + & where we set
B.5 3D INSTANCE SEGMENTATION
For 3D instance segmentation, we retrain Mask3D (Schult et al.) 2023) our ScanWCF dataset, following the training procedure that they use for ScanNet. Training for 600 epochs with batch size of 3 on scenes with 2 cm voxelization takes approximately 4 days on single RTX 4090 GPU. We note that the instance segmentations produced by Mask3D are only used during inference. During the training phase of our completion model the ground truth partial object instances.
B.6 MESH RECONSTRUCTION
Meshes our object completions are only reconstructed at inference time. For this, we directly use NKSR's (Huang al., (202)) pre-trained kitchen-sink model, which has been jointly trained across several object and scene scale datasets.
C SCANWCF DATASET
Our proposed Scan WCF dataset is composed of data gathered from the ShapeNet (Chang et al, 2015). , ScanNet (Dai et al. 2017), Scan2CAD (Avetisyan et al. (2119)), and ScanARCW (Li al, 2023a) datasets. All data was obtained directly through publicly available download links on their websites and permission to use the data granted for datasets which required it.
The Scan WCF dataset consists of 1202 scenes, which are based off scenes from the ScanNet dataset. Our dataset does not contain all 1513 scans from ScanNet relies the background meshes from the ScanARCW dataset, which are only provided for 1274 of the scenes. Additionally, the scenes in our dataset must pass manual verification process of which 72 of the 1274 scenes failed. Despite our dataset not containing all the scenes from ScanNet, we reuse the scene id train/test split from the original dataset, resulting in 946 training scenes and 246 test scenes. Objects from 34 different categories from ShapeNet are present across all the scenes in our dataset. While we train each method with all the objects present the training scenes, we only evaluate the instance scene completion task on the 13 categories which had more than 150 training examples present across all the scenes. We chose this as we observed poor instance segmentation performance across methods on categories which contained small number of training examples.
In the following sections, we describe the data generation process for our proposed ScanWCF dataset, which is both Watertight and Collision Free (hence the name ScanWCF).
C.I PRE-PROCESSING FOR WATERTIGHTNESS
The objects provided the ShapeNet dataset are not guaranteed to watertight, making difficult reason about collisions between objects. To address this, process ShapeNet objects for wa-
14

---

Published as a conference paper at ICLR 2025
tertightness using the method proposed by Wang al. (2022). This enables to obtain the signed distance of any point with respect to the object mesh. making collision checking easy. To reason about collisions with scene boundaries (e.g.. walls, floor, ceiling), from the ScanARCW dataset. While the background mesh for each scene aligns with the corre- sponding scan from ScanNet, some of the background meshes have holes in them where window or door meshes would be placed. Unlike ScanARCW, we leave out the door and window meshes and instead fill the holes in the background meshes make them watertight.
the background meshes
C.2 OPTIMIZING SCENE LAYOUT
Our goal is to be able to produce a scene layout containing meshes which closely align with objects in a ScanNet scene while being free of collisions. In order to do so, we begin by initializing our scene as the empty watertight background mesh which is already aligned with real scan from ScanNet. We then proceed to place one object into the scene at a time, optimizing for both the pose and scale of the object. We make use of the Scan2CAD dataset for deciding which ground truth meshes from ShapeNet to place in the scene, as the dataset has already detected objects in the real ScanNet scenes and matched them their closest mesh in ShapeNet. Additionally, we initialize each object's pose and scale using the Scan2CAD annotations as they have roughly aligned the synthetic meshes to the real scans, Then we optimize the pose and scale of each object such that the following criteria are best met:
Alignment with partial sean. We minimize the Chamfer Distanee between points sampled the ground truth mesh and the partial object it was matched to within the real ScanNet scene.
2. Minimize amount of collisions. We uniformly sample points on the object mesh and com- pute their signed distance to the scene mesh (both objects and background). We penalize points which fall inside another object or outside the scene boundaries.
3. Minimize large changes in scale. We penalize large deviations in object scale from their initialized value. This prevents the optimization from shrinking the object scale by large amounts to reduce collisions.
4. Minimize floating objects. Minimizing the amount of collisions while penalizing changes in scale can lead to the optimization producing objects floating in space. To help prevent this, we sample points on the bottom of the object mesh and compute their signed distance to the scene mesh. We penalize values which deviate from O, which signifies the object is not resting on surface.
After an object's pose and scale has been optimized, it is added to the ground truth scene mesh to provide further constraints for the objects which will later be optimized.
C.3 MANUAL VERIFICATION
Not all scene layouts guaranteed to be free of collisions upon the optimization phase finishing. Therefore, after optimizing for the scene layout, we manually verify that the scene is plausible. no objects in collision with each other and the scene looks plausible (c.g., no floating objects), we include the scene in our dataset. If the scene includes some objects which are in collision or implausible,
manually correct the issue and include the scene in our dataset. If too many collisions still exist after our optimization step, we simply disregard the scene from being included in the dataset.
C.4 PRODUCING ALIGNED PARTIAL SCANS
After the manual verification phase, we have a set of scenes which serve the ground truth instance completions for our dataset and now need to generate partial scans which are aligned to them. To generate partial scans of the scene, we render depth maps from different viewpoints and backproject them back into 3D. For a particular scene id, we make use of the camera extrinsics and intrinsics from the corresponding scene id in the ScanNet dataset to render 2D information with. Using Py D (Ravi et al., 2020). we render depth maps, 2D instance segmentations, and surface normal maps of the scene from each viewpoint. We share some example renderings of scene in Figure Sa Upon
15

---

Published as a conference paper at ICLR 2025
(b)
Figure 8: (a) We render our ground truth scene meshes into depth maps, instance segmentation maps, and surface normal maps from multiple views. (b) Each scene contains a ground truth labeled scene mesh and an aligned and labeled partial scan constructed from our multi-view renderings of the scene.
rendering 2D information, we select 10 viewpoints per scene and backproject their 2D information into world coordinates, fusing together the different point clouds to produce partial scan as shown in Figure 8b While backprojecting the depth map of each viewpoint into point cloud, we also estimate surface normals and orient them such that they are correctly oriented towards the camera. To keep the number of points in the scene reasonable. subsampling with a em resolution. Finally, To increase the amount of training/test data, for each scene, choose generate partial scans using different subset of 10 ScanNet camera poses.
resample each partial scan using grid
16

---

Published as a conference paper at ICLR 2025
D METRICS
D.1 SCENE COMPLETION METRICS
For the scene completion task, we have removed the need predict instance proposals from each method. Instead we provide each method with the ground truth instance segmentations so that each network is only responsible for completing the partial object. In this setting, false positive detections/proposals are not possible and therefore we do not have to rely on the use of mean Average Precision (mAP) as way to evaluate completion quality. Instead we can evaluate the completion quality of each approach using common metrics from the point cloud completion literature. Each metric used the scene completion task defined in the following sections.
D.1.1
CHAMFER DISTANCE (CD)
The Chamfer Distance between two point clouds P E R. and Q E RMM Mx can be defined as:
dD (P.Q)Q = Σ min VEQ
x
yll
min PEP
x
(5)
Q
We use Chamfer Distance as a measure of completion quality. For evaluation, we sample 2048 points per predicted object and concatenate each object's point set into single scene point cloud. We do the same procedure for the ground truth objects, uniformly sampling the 2048 points from each object mesh. Chamfer Distance is then measured between the predicted scene point cloud and ground truth scene point cloud using Equation 5 Numbers reported in tables have been scaled by factor of 10a.
D.1.2 ONE-SIDED CHAMFER DISTANCE
Equation meassrrs the bi-directional (or symmetric) Chamfer Distance. To evaluate how well the completion respects the partial input, we instead measure the One-sided Chamfer Distance. The One-sided Chamfer Distance between two point clouds P E Rx3 and Q RMx3 can be defined as:
(CP PEPPEE PEP
E min - y1
VEQ
We measure the One-sided Chamfer Distance between the 2 cm resolution partial scan excluding the background (i.e., points on the walls, floor, ceiling) and our predicted scene point cloud. Numbers reported in tables have been scaled by factor of 10
D.1.3 UNIDIRECTIONAL HAUSDORFF DISTANCE (UHD)
One-sided Chamfer Distance measures the average deviation of the partial reconstruction from the partial input. We additionally measure the maximum deviation of the partial reconstruction from the partial input using the Unidirectional Hausdorff Distance (UHD). The Unidirectional Hausdorff Distance between point clouds P Nxxx x3 and Q M x3 can be defined as:
UHD (P.(P.C Q) max min #EP y
1112
Similar to One-Sided Chamfer Distance, we measure UHD between the 2 em resolution partial scan with background removed and predicted point cloud. Numbers reported in tables have been scaled by a factor of 1
D.1.4 COLLISION METRIC (COL)
We measure completion plausibility by how badly predicted completions collide with each other or the scene boundaries. Let the watertight background mesh (i.e., walls, floor, ceiling) be denoted as and the set of predicted watertight object meshes be denoted as {M,,..., M,. We define SDF(M,p) p) represent the signed distance of arbitrary 3D point surface of mesh M. where (DF < 0 implies the point p falls inside mesh M. Addi- tionally, we denote set of point clouds which reconstructed meshes from
its closest point the
17

---

Published as a conference paper at ICLR 2025
(for RfD-Net and DIMR sample 2048 points uniformly from mesh M, to produce point cloud completion C₁). Each object point cloud C, is composed of a set 3D points, where we denote a point being in the point cloud C. Now can define the collision metric for scene
COL
Σ PEC;
min(O, -SDF(B,P)) + Σ min(O,SDF(Ms;P))
If points in the scene completion violating scene constraints (i.e., penetrating into other ob- jects or the scene boundaries), the collision metric is O. Otherwise, the collision metric is equal the average of how far each point has penetrated into an object extended outside the scene boundaries.
D.2 INSTANCE SCENE COMPLETION METRICS
To evaluate our proposed approach on the joint task of instance segmentation and object completion, the metrics and setup proposed by Tang et al. (2022). In particular, compute the 3D detection mean Average Precision (mAP) for each metric defined the following sections. For computing average precision, the PASCAL VOC 2007 1-point interpolation method is used.
D.2.1 INTERSECTION OVER UNION (IoU)
Intersection over Union is voxel-based approach for evaluating completion quality. It measures the predicted completion's voxel occupancy against the ground truth completion's voxel occupancy. To compute it, both predicted meshes and ground truth meshes are first voxelized with fixed voxel size of 4.7 em. Then Intersection over Union is computed the Volume of overlap with regards to voxel occupancy.
Volume union
D.2.2 CHAMFER DISTANCE (CD)
Chamfer Distance provides a point-based evaluation of completion quality. measures the distance from points sampled on the surface of the predicted mesh to points sampled on the ground truth mesh. To generate point sets, we uniformly sample 4096 points from both the predicted mesh and the ground truth mesh. Chamfer Distance is then computed using Equation
D.2.3 LIGHT FIELD DISTANCE (LFD)
Light Field Distance is visual similarity metric for meshes. The main idea is that if the predicted complete mesh is similar to the ground truth mesh then should look similar from all viewpoints. To compute LFD, each predicted mesh and ground truth mesh is rendered into 2D images from multiple viewpoints and encoded into a light field descriptor that is used for measuring distances between meshes. We refer readers to the work by Chen et al. (2003) for description of the light field descriptor.
D.2.4 POINT COVERAGE RATIO (PCR)
Point Coverage Ratio measures how well the completed mesh aligns with the partial input. The Point Coverage Ratio between a point set P E R x3 and mesh M can be defined as:
PCR(P,M)
Σ
1 M[as(e,M))Tppr
P
where I is the indicator function, dist(æ,, M) the distance from the point p to the surface of mesh M, and Tper is a distance threshold (we set Tper 0.047 for evaluation).
D.3 SURFACE NORMAL METRICS
In section E.S. we evaluate the quality of our surface normal predictions needed to reconstruct a mesh. Rather than directly evaluate accuracy of the predicted surface normals, evaluate the
18

---

Published as a conference paper at ICLR 2025
quality of the reconstructed meshes produced using NKSR (Huang et al. 2023) with our point cloud and surface normals. To evaluate object mesh quality, we use the metrics and setup proposed by Park et al. 2019).
D.3.1 CHAMFER DISTANCE (CD)
Chamfer Distance is used to evaluate overall shape quality. In particular, we sample 30, 000 points on both the predicted and ground truth meshes and compute the Chamfer Distance between the two point sets using Equation
Here a lower value indicates a better match to the ground truth shape.
D.3.2 MESH COMPLETION
Mesh completion evaluates how well the ground truth surface is covered by the predicted mesh reconstruction. In particular, we compute the percent of points sampled from the ground truth surface whose distance to the predicted mesh reconstruction Teomp 0.01 following Park al. (2019)). We compute the mesh completion metrie over 1000 points uniformly sampled from the ground truth mesh. A higher score indicates better coverage of the ground truth surface by the predicted mesh reconstruction.
within a threshold Toomp (we set
D.3.3 MESH ACCURACY
Mesh accuracy evaluates how close points on the predicted mesh surface are to the surface of the ground truth mesh. In particular, we find the minimum distance threshold Tace such that 90% of points sampled from the surface of the predicted mesh are within distance Taoe to the ground truth surface. To compute mesh accuracy, we sample 1000 points from the predicted mesh reconstruction and compute each points distance to the ground truth surface. A lower score indicates better accuracy the ground mesh.
E RESULTS
In this section share more results of our scene completion method.
E. RUNTIME
On an RTX 4090 GPU, our completion model takes an average of 0.104 seconds to complete a scene containing an average of 9 objects in it. In other words, our method can complete about 86 objects per second.
E.2 INSTANCE SCENE COMPLETION
In Table 5 and Table 6 share per class Average Precision (AP) scores for the Intersection over Union (IoU) metric at thresholds 0.25 and 0.5, respectively. Note that at both thresholds our method outperforms RiD-Net and DIMR across almost all categories. In particular, observe significant gap in the performance of our method compared to previous approaches when evaluating at the more difficult threshold (T..le
In Table 7 and 8 we show per class AP scores for the Chamfer Distance (CD) metric at thresh- old 0.1 and 0.047, respectively. Both RiD-Net and DIMR suffer large drops in performances when evaluating on the more difficult threshold (Table 8 On the other hand, the drops in performance, if any, for our method are much smaller.
Along with CD, we evaluate completion quality using the Light Field Distance (LFD) metric which measures visual similarity between meshes. In Table 9 and Table [0 we see that RID-Net and DIMR behave similarly to how they do with CD and IoU, achieving decent performance easier thresholds but very low scores at the more difficult thresholds. This trend occurring across all three only produce completions which somewhat simi-
metrics suggests that previous approaches lar in shape to the ground truth objects. On the other hand, we see our method obtains a much higher LFD score the more challenging threshold, which is similar to how perform CD and loU
19

---

Published as a conference paper at ICLR 2025
compared to previous works, suggesting that our method produces completions that more accurately represent the geometry of the objects in the scene.
share per class AP scores To evaluate partial reconstruction quality, in Table I1 and Table 12
on the Point Coverage Ratio (PCR) metric. Our method observes almost no drop in performance between the easier and more difficult threshold across all categories, demonstrating that com- pletions align with the partial scans with high fidelity.
Finally, in Figure 9 and Figure 10 share more qualitative comparisons against previous works RfD-Net and DIMR on the instance scene completion task.
E.3 SCENE COMPLETION
We share qualitative results when the ground truth instances have been supplied to each method. Despite having the ground truth instance information, RfD-Net still produces low-quality completions of chairs and tables while DIMR fails to complete or even reconstruct the observed inputs in many cases. Additionally, we see RID-Net tends to produce many collisions between completions. On the other hand, our approach respects the partial input well and completes the fine-grained geometry that missing from the scans while avoiding collisions between predictions.
In Figure 11
E.4 ANALYZING THE EFFECTS OF OBJECT SCAN INCOMPLETENESS
In Figure 12 we further investigate how the completion quality our method varies under different
levels of incompleteness in the partial input. In Figure 12a we present histogram breaking down the object instances the test set of our dataset by how much of the complete shape present the
partial object scan. We find that majority of our partial object scans contain between 30 60% of complete geometry before being input to our completion model. the average completion metries for various methods for each bin in our histogram. We find that method outperforms the baseline approaches at each completeness level by large margin, which is consistent with the large gap in performance observed in main results shown in Table T and Table However, we do observe that RiD-Net and DIMR do outperform our method for extremely sparse imputs (e.g.. when only 10% of the object is present in the partial scan).
Figures 12b 12d we plot
E.5 ANALYZING THE EFFECTS OF IMPERFECT INSTANCE SEGMENTATION
In Figure 13 visualize example completions when there errors present in the instance segmentation predictions produced by Mask3D. In the top four rows, we show examples where Mask3D produced segmentations that missed large important parts of the partial object instance. The regions missed by Mask3D contain important cues for the true geometry and size of the object, and the lack of this information leads our model produce a completion which is different from the ground truth completion. In the bottom two rows, we show examples where Mask3D incorrectly segments two objects that are side by side as a single object. In this scenario, our completion model has no knowledge that there are actually two objects present and instead completes both objects together as it were one.
E.6 GENERALIZATION TO REAL SCANS
In Figure 14 we share some example completions which demonstrate our methods ability to gen- eralize to partial objects from real scans in the ScanNet dataset. While we sample object instances from categories we have trained on (e.g., chair, table, trash bin), all the object instances shown are completely novel the ScanNet object instances have no overlap with objects in the ShapeNet dataset. Moreover, the partial scans from ScanNet are considerably less clean than the partial scans from our dataset, yet method is still able produce plausible completions of the missing regions of the objects.
E.7 ABLATIONS
In Figure 15 we present a qualitative comparison our scene completion model with and without pre-training. While our method can outperform previous approaches without pre-training the object
20

---

Published as a conference paper at ICLR 2025
completion model, find that pre-training is important for reasoning about the missing geometry when entire parts of an object are missing from the partial scan. In the first scene of Figure [5] completion model is able to represent the thin legs of the chair
see that without pre-traie pre-training when some of the legs are observed in the partial scan (e.g., cyan and green chairs), but produces a solid circular base instead of thin legs when the entire base of the chair missing (e.g., blue, purple, and pink chairs). On the other hand, when we pre-train our object completion model we are able to recover the individual legs of the chair despite not observing any part of due to the model having prior over objects which it can fall back on when needing to hallucinate the missing structures of largely occluded objects. Furthermore, we find that pre-training helps clean up noisy predictions such as the dark blue chair in the third scene of Figure 15
E.8 SURFACE NORMAL ESTIMATION
we share qualitative comparison of our completions and estimated surface normals compared to the ground truth meshes and surface normals. We find that method does good job at recovering the complete geometry of the scene even for small details such as the handles on drawers of the bathroom sink in the first scene the thin cross bars the legs of the chairs in the last scene.
In Figure 16
For mesh reconstruction, NKSR (Huang al. 2023) typically uses PCA-based plane fitting tech- nique for estimating normals of point cloud. In Table 13 and Figure 17 we show that normals es- timated using PCA-based plane fitting suffer from poor reconstruction quality on our completions. We find that varying the neighborhood size used in plane fitting does not seem to help improve reconstructions. Instead the normals produced by our normal estimation module lead to signif- icantly better reconstructions, outperforming PCA-based normal estimation across all metrics. In Figure [7] we show that PCA-based plane fitting suffers from inaccurate normal estimation in the presence of noise in the completion or thin structures such as the legs of chairs, leading to arti- facts the reconstructions. Alternatively, our estimated normals are oriented correctly despite noise being present the completion containing thin structures, producing much better reconstructions with NKSR.
E.9 FAILURE CASES
In Figure 18 we share some example failure cases of our scene completion model. We find that our method occasionally suffers from collisions between predictions when two objects are up against each other but there are no scene constraint points suggesting a separation between the two objects. An example of this can be seen in the left most scene of Figure 18] where the partial scan (black points) contains no information on how far the blue drawer should extend to the right, causing it to overextend into the yellow drawer. While our object completion model does include information about the scene when producing a completion, it only considers observed information present in the partial scan. This means our model cannot reason about avoiding collisions in regions of space that will be filled by another object's 's completion.
Table 5: loU@0.25 Per class AP scores for mesh quality based on occupancy against ground truth voxels using Intersection over Union (loU) with threshold of 0.25. denotes our scene completion model results without pre-training.
Lamp
Big
Display
RIT-NNet
22.79
45.75
41
30:37
13,95
31.21
DIMR
8.00
Durs
70.74
86.77
66.08
61.34
72.33
77.89
11.06
63.40
49. 05.56
61.97
78.76
62.05
43,90
Durs
71.71
86.86
65.84
68.97
72,33
77.89
11.06
63.40
49.79
63.30
79,31
62.35
43.39
21

---

Published as a conference paper at ICLR 2025
Table 6: I.5 Per class AP scores for mesh quality based on occupancy against ground truth voxels using Intersection over Union (IoU) with threshold of 0.5. i denotes our scene completion model results without pre-training.
Table
Chair
Bookshelf
Sofa
Lamp
Irash Bun
Fice Cabinet
Bag
Cabinet
Bed
Display
Bathiub
Printer
RID Net
1.81
0.66
2.19
0.09
2.27
32.08
0.65
29.55
3.03
0.20
11.76
9.85
0.30
DIMR
11.13
6.57
11.02
1.81
0.00
34.45
0.00
0.00
0.57
0.81
31.72
14.56
0.00
72.16
59.33
10.39
53.98
31,69
64.45
45.39
51.34
10.05
Table 7: CD@0.1 Per class AP scores for mesh quality based on distances between mesh surfaces using Chamfer Distance (CD) with threshold of 0.1, i denotes scene completion model results
without pre-training.
Table
Chuir
Bookshelf
Sofa
Lamp
Trach Din
File Cabinei
Bag
Cabinel
Bed
Display
Bathtub
Printer
RID-Net
58.39
88.25
62.66
49.33
44.71
74,80
23.30
69.84
57.30
63.28
85.74
74.44
49.78
DIMR
70.69
89.32
58.53
50.37
0.00
64.05
0.00
0.00
36.30
59.82
50.31
18.18
7.57
71.25
86.89
72.33
77.89
11.06
79.31
Oues
71.49
86.92
37.00
61.46
72.33
77.89
11.05
63.40
49.35
62.91
79,95
02:25
43.90
Table 8: CD@0.047 Per class AP scores for mesh quality based on distances between mesh surfaces
using Chamfer Distance (CD) with threshold of 0.047. denotes our scene completion model results without pre-training.
Lamp
Bag
Cabinee
Display
RID-Net
14.55
36.077
25.63
68.78
11.25
69.12
22.54
56.85
DIMR
17.66
17.39
Durs
57.46
85.64
46.95
45.94
70.27
77.08
10.39
63.40
40.54
38.89
68.36
61.93
43.90
Oues
59.36
86.00
48.55
45.71
70.27
77.00
10.39
63.40
41.74
51.06
77.20
61.93
36.36
Table 9: 5500 Per class AP scores for mesh quality based on visual appearance using Light Field Distance (LFD) with threshold of 5000. t denotes our scene completion model results without pre-training.
Table
Chair
Bookshelf
Sofa
Lamp
Irash Hun
File Cabinet
Bag
Cabinee
Bed.
Display
Bathtub
Printer
RID-Net
12,55
4.73
8.00
27.65
DIMR
24.76
10.30
61.37
53.59
8.00
50.59
0.00
01.00
44.69
40.19
23.20
26.65
17.53
Ours
50.12
59.77
46.54
60.37
36.57
73,07
18.19
63.40
50.55
59.15
54.43
60.81
43.39
64.79
27.75
18.71
52.14
61.15
Table LFD 2500 Per class AP scores for mesh quality based on visual appearance using Light Field Distance (LFD) with threshold of 2500. denotes scene completion model results without pre-training.
Table
Chair
Boakshellf
Sofa
Lamp
Triss Bus
File Cabinet
Bag
Cabinet
Bed
Display
Bathtub
Printer
RID-Nee
1.81
0.43
30.54
4.85
0.00
29.24
22.94
26.93
33.67
12.73
0.12
24.09
3.09
DIMR
0.87
3,03
31.68
13.95
0.00
5.82
(1.00
01.00
32.85
7.38
0.16
17.22
1.52
5,05
28.72
32.47
23.35
3.64
12.25
27.30
39.21
28.76
32.17
38,55
Table 11: PCR @0.5 Per class AP scores for point-to-mesh mapping quality using point coverage ratio (PCR). denotes our scene completion model results without pre-training
Table
Chair
Booksbelf
Sofa
Lamp
Trash Din
File Cabine!
Bag
Cabinet
Bed
Display
Bathtub
Printer
RID-Net
47.76
72.11
22.69
79.44
DIMR
76.87
80.67
64.08
59.07
8.00
63.0 15
0.00
0.00
37.22
70.70
77.33
25.65
7.27
Durs
23.31
86.97
66.42
68.98
72.33
77.89
11.06
63.40
50.68
70.94
N0.37
62.17
43,50
86.97
66.38
72.33
77.89
11.06
70.94
Table 12: PCR@0.75 Per class AP scores for point-to-mesh mapping quality using point coverage ratio (PCR). denotes our scene completion model results without pre-training.
Lamp
Big
Display
RIT-NNe 34.72
59.53
30.56
15.64
25.32
58.03
13,64
67.85
47.60
64.10
DIMR
8.00
Durs
70.80
86.33
65.25
68.63
72.33
77.89
11.06
63.16
49.97
63.23
80.37
62.17
43,90
Durs
70.32
86.13
65.55
68.63
72,33
77.89
11.06
63.15
49.90
63.44
N0.37
62.17
43,80
22

---

Published as a conference paper at ICLR 2025
Partial Scan
RiD-Net
DIMR
Ours
Ground Truth
Figure 9: Qualitative comparison the instance completion task.
Table 13: Evaluation of mesh reconstruction quality using our predicted surface normals vs. PCA- based normal estimation.
CD (1)
Mesh Comp. (1)
Mesh Acc, (L)
PCA (k=16)
28.83
0.327
0.062
PCA (k=32)
28.58
0.331
0.063
PCA (k=48)
28.51
0.321
0.062
Ours
21.12
0.508
0.046
23

---

Published as a conference paper at ICLR 2025
Partial Scan
RiD-Net
DIMR
Ours
Ground Truth
Figure 10: More qualitative comparisons on the instance scene completion task.
24

---

Published as a conference paper at ICLR 2025
RID-Net
DIMR
Ours
Figure 11: Qualitative results on the scene completion task when ground truth instance information provided.
25

---

Published as a conference paper at ICLR 2025
Iwparr
(a)
(b)
ESS
240
C4
completeens tataickpec
DUNITN)
(C)
(d)
Figure 12: (a) Histogram breaking down the objects in our test dataset based on how much of the complete shape present in the partial object scan. (b) Comparison of average Chamfer Distance for various methods for each bin in our histogram. (c) Comparison of average One-Sided Chamfer Distance for various methods for each bin our histogram.
(d) Comparison of average Unidirec- tional Hausdorff Distance for various methods for each bin in our histogram. We note that each data point is an average over all the partial objects that fall into a bin our histogram (e.g., the data points at x=1.0 are really the average metric completions over all partial objects which fell into histogram bin [0.9, 1.0)).
26

---

Published as a conference paper at ICLR 2025
E
GT Instance
MaskID Instance Pred.
Our Completion
GT Completion
Figure 13: Example completions imperfect instance segmentation predictions. Errors in the in- stance segmentations produced by Mask3D can lead to our method producing incorrect completions.
27

---

Published as a conference paper at ICLR 2025
Figure 14: Example completions when generalizing to partial objects from real scans in the ScanNet dataset (Dai et al. 2017).
28

---

Published as a conference paper at ICLR 2025
W/out pre- ge-trraiiing
W/ pre- training
Figure 15: Qualitative comparison of our scene completion model with and without pre- mreitraining
29

---

Published as a conference paper at ICLR 2025
88
Partial Scan
Completion & Normals
Ground Truth
Figure 16: Qualitative results of our predicted completions and surface normals.
30

---

Published as a conference paper at ICLR 2025
Partial Input
GT Mesh
PCA Normals
Recon. (PCA)
Our Normals
Recon. (Ours)
Figure 17: Comparison of reconstructing meshes from our completions with NKSR (Huang et al.
2023 using PCA-based estimated normals vs. our estimated normals.
Figure 18: Example failure cases of scene completion method.
31


Reviews:
[{"review_id": "c5nOYuzfln", "reviewer": "Reviewer_M7dZ", "summary": "This paper presents a scene-level instance completion method based on point clouds. The main contributions are two-fold. First, a more sophisticated seed generator is used to obtain patch seeds. It predicts the offsets to the centers of the objects and leverages scene constraints to generates more reasonable seeds. Second, the authors construct a new dataset for scene instance completion task, which contains watertight and collision-free ground truths. The authors conduct comprehensive experiments and ablation studies to demonstrate the effectiveness of the method.", "questions": "Please address the questions in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The illustrations are clear and help understand the paper.\n3. The paper has new dataset contributions.\n4. The experiment results are promising.", "weaknesses": "1. In Line 208, the authors claims that the local attention-based upsampler is worse than the global attention-based method when the objects are not canonical. I cannot see the logic behind. Maybe this is because the proposed method uses VI-PointConv in the encoder? I think the authors should provide more detailed investigation of this.\n2. Some important details are missing.\n- In Line 203, how does the global shape descriptor generated?\n- In Line 206, why are the seeds called the patch seeds? There should be an explanation here.\n- In Line 237, how is the tranposed convolution used?\n3. How does the proposed method generalize to unknown objects during training?\n4. The organization of the paper should be improved. For example, scene-aware object completion (Sec 3.3) is part of the seed generator (Sec 3.2), and describing them separately makes the paper somehow confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J4n746cDWN", "reviewer": "Reviewer_TJHm", "summary": "This paper argue that existing instance scene completion methods may neglect scene constraints when doing instance completion. To address these issues, they propose a point cloud-based instance completion model that utilizes scene constraints through a cross-attention mechanism for improved object completion at any scale or pose. Besides they introduce a new synthetic dataset, ScanWCF, to evaluate instance scene completion in indoor environments. Experiments show that their method outperforms existing techniques in terms of accuracy and plausibility.", "questions": "I am curious that if there are no scene constraints, how would the object completion module perform against the baselines?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the major contribution of the seed generator converted on top of SeedFormer, that allows the generator to use information from the entire scene to predict the center of the object before producing seed coordinates as offsets of the object center.\n\nAnother is the incorporation of a sparse set of scene constraints for seed generator and completion, providing scene context (e.g., other observed surfaces, free space, occluded space).", "weaknesses": "I think this is a good paper with sound method and formulation. However, from the technical contribution's perspective, this paper still follows the same \"top-down\" architecture as in RfDNet, and the performance's boost against baselines comes from the better quality of each individual module. The module this paper is mostly contributing is the object points completion module that completes object pointcloud from partial points given by leveraging the scene context/constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dbdQE7PYtH", "reviewer": "Reviewer_eRJX", "summary": "This paper proposed an instance scene completion method for partial point cloud which consists of 3 stages: instance segmentation stage which extract each object from partial point cloud; object completion stage which predicts both complete shape and surface normals for each object; and mesh reconstruction stage which reconstruct scene meshes from point clouds. The focus of this paper is to propose a set of improvement and refinement modules to improve the object completion network. Furthermore, a new dataset for instance scene completion in indoor scenes has been proposed which has better ground truth labels than existing ScanARCW dataset. Experiment on the newly proposed dataset shows that the proposed method achieves SOTA performance on instance scene completion task.", "questions": "Since existing instance scene completion method jointly train instance segmentation & completion model, while the proposed method uses existing Mask3D method for instance segmentation separately, is there any experiment comparison on the performance of instance segmentation? I may want to see whether the biggest peformance improvement is due to the use of Mask3D?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper is a good practice to build a instance scene completion framework based on the core module of encoder-decoder object completion. There are several design improvements over existing work in the proposed method:\n1. Use VI-PointConv to replace PointConv in the partial encoder module in object completion stage.\n2. Use object center and offset prediction MLP to generate aligned seed patch in the Seed Generator module in object completion stage.\n3. Incorporate the scene constraints into the Seed Generator module through cross-attention\n4. Use global attention in upsample layers to improve the Coarse-to-Fine Decoder module in object completion stage.\n5. Use surface normal prediction module which enables mesh reconstruction using existing NKSR method\nAll the above improvements have been verified in ablation study\n\nExperiment shows that the proposed method significantly outperforms existing instance scene completion methods RfD-Net and DIMR (table 1&2)\n\nThe paper also proposed a refined dataset 'ScanWCF' which is declared to have 'Watertight and Collision Free' ground-truth, this could benefit future research on instance scene completion task", "weaknesses": "There are some concerns on the novelty and effectiveness of the declared main contributions, as the paper said, there are 3 main contributions, the first contribution is 'a novel object-level completion model which is robust to the scale and pose of objects found within scenes', but actually, the whole framework is to deal with instance scene completion, and in instance scene completion, there is no need to constrain the scale and pose of the objects in the scene. the second contribution is 'integrate a sparse set of scene constraints into our model to provide our object completions with scene context', while the corresponding ablation study in table 4 shows limited improvement on UHD and %COL metric. I think the presentation of this paper could be further improved to make the paper stronger\n\nIn the ablation study which focus on the object completion task (table 3), the performance improvement of each proposed module is relatively small and not very impressive compared with that on the scene completion task (Table 1&2), furthermore, each module is added separately, what is the performance with all modules included?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zpFKFikw25", "reviewer": "Reviewer_gWgp", "summary": "This work leverages joint scene and object information for object point cloud completion in the scene, which is a meaningful and important task. In order to boost completion, the work investigate both object-centric learning and scene occupancy information to guide the completion. In detail, it uses object transformers to generate patch seed in object-centric coordinate with an additional object coordinate, and use scene transformers to represent scene occupancy information. Then, a cross attention is used to combine the information for completion.\n\nThe experiments suggest significant improvement using the proposed method.", "questions": "Check the weakness part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The authors address the point cloud completion task in the scene, which is a fundamental problem in 3D vision. They offer insights that the key for completion is to extract object-centric information in the scene, and combine useful global scanning information such as scene visibility/occupancy.\n\nAccordingly, they provide a network that allow both insights to be implemented.\n\nThe idea is clear and the method is solid. Besides, I believe the design shows sufficient technical novelty that reaches the conference standard.", "weaknesses": "It may be worth discussing the limitation and quality of point cloud completion using the proposed method under different scenario, e.g., what is the requirement for noise/incompletion ratio for the method to succeed? Can it complete unseen objects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)