PatchComplete: Learning Multi-Resolution Patch Priors for 3D Shape Completion on Unseen Categories
Authors: Yuchen Rao
Yinyu Nie
Angela Dai
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3d shape completion
3d reconstruction
zero-shot 3d reconstruction

Abstract: While 3D shape representations enable powerful reasoning in many visual and perception applications, learning  3D shape priors tends to be constrained to the specific categories trained on, leading to an inefficient learning process, particularly for general applications with unseen categories. Thus, we propose PatchComplete, which learns effective shape priors based on multi-resolution local patches, which are often more general than full shapes (e.g., chairs and tables often both share legs) and thus enable geometric reasoning about unseen class categories. To learn these shared substructures, we learn multi-resolution patch priors across all train categories, which are then associated to input partial shape observations by attention across the patch priors, and finally decoded into a complete shape reconstruction. Such patch-based priors avoid overfitting to specific train categories and enable reconstruction on entirely unseen categories at test time. We demonstrate the effectiveness of our approach on synthetic ShapeNet data as well as challenging real-scanned objects from ScanNet, which include noise and clutter, improving over state of the art in novel-category shape completion by 19.3% in chamfer distance on ShapeNet, and 9.0% for ScanNet.

Patch Complet: Learning Multi-Resolution Patch Priors for 3D Shape Completion on Unseen Categories
Yuchen Rao
Yinyu Nie
Angela Dai
Technical University of Munich
(yuchen.rag rao, yinyu.nie, angela. @tum. de
Abstract
While 3D shape representations enable powerful reasoning in many visual and perception applications, learning 3D shape priors tends to be constrained to the specific categories trained on, leading to inefficient learning process, particularly for general applications with unseen categories. Thus, we propose PatchConplete, which learns effective shape priors based on multi-resolution local patches, which are often more general than full shapes (e.g., chairs and tables often both share legs) and thus enable geometric reasoning about unseen class categories. To learn these shared substructures, we learn multi-resolution patch priors across all train categories, which then associated to input partial shape observations by attention across the patch priors, and finally decoded into complete shape reconstruction. Such patch-based priors avoid overfitting specific train categories and enable reconstruction on entirely unseen categories at test time. We demonstrate the effectiveness of our approach on synthetic ShapeNet data as well as challenging real-scanned objects from ScanNet, which include noise and clutter, improving over state of the art in novel-category shape completion by 19.3% in chamfer distance on ShapeNet, and 9.0% for ScanNet.
Introduction
The prevalence of commodity RGB-D sensors (e.g., Intel RealSense, Microsoft Kinect, iPhone, etc.) has enabled significant progress in 3D reconstruction, achieving impressive tracking quality [25, 17, 27, 8, 40, 12] and even large-scale reconstructed datasets [10, 4]. Unfortunately, 3D scanned reconstructions remain limited in geometric quality due to clutter, noise, and incompleteness (e.g., g. as in the objects in Figure 1). Understanding complete object structures is fundamental towards constructing effective 3D representations, that can then be used to fuel many applications in robotic perception, mixed reality, content creation, and more.
Recently, significant progress has been made in shape reconstruction, across a variety 3D rep- resentations, including voxels [9, 13, 34], points [15, 43], meshes [39, 11, 32]. and implicit field representations [28, 21, 14, 18, 31, 3, 16, 33]. However, these methods tend to rely heavily on strong synthetic supervision, producing impressive reconstructions similar train class categories but strug- gling to generalize to unseen categories. This leads to an expensive compute and data requirement in adapting new objects in different scenarios, whose class categories may not necessarily have been seen during training and so must be re-trained or fine-tuned for.
In order to encourage more generalizable 3D feature learning to represent shape characteristics, we observe that while different class categories may have very different global structures, local geometric structures are often shared (e.g., a long, thin structure could represent chair leg, table leg, lamp
Source code available here.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Training (Seen) Categories
Testing (Novel) Categories
Learnable MustitResoottion Priors
Input
PatchCompicte
GT
Shape Priors
8 Patch Priors
4 Patch Priors
PatchComplete learns strong local priors for 3D shape completion by constructing multi- resolution patch priors from train shapes, which can then be applied for effective shape completion on unseen categories.
Figure 1:
rod, etc.). We thus propose to learn a set of multi-resolution patch-based priors that captures such shared local substructures across the training set of shapes, which can be applied to shapes outside of the train set of categories. Our local patch-based priors can thus capture shared local structures, across different resolutions, that enable effective shape completion novel class categories of not only synthetic data but also challenging real-world observations with noise and clutter.
We propose PatchComplete, which first learns patch priors for shape completion by correlating regions of observed partial inputs the learned patch priors through an attention- based association, and decoding to reconstruct complete shape. These patch priors are learned at different resolutions to encompass potentially different sizes of local substructures; we then learn fuse the multi-resolution priors together to reconstruct the output complete shape. This enables learning generalizable local 3D priors that facilitate effective shape completion even for unseen categories, outperforming state of the art in synthetic and real-world observations by 19.3% and 9.0% on Chamfer Distance.
In summary, our contributions are:
We propose generalizable 3D shape priors by learning patch-based priors that characterize shared local substructures that can be associated with input observations by cross-attention. This inter- mediate representation preserves structure explicitly, and can be effectively leveraged to compose complete shapes for entirely unseen categories.
We design a multi-resolution fusion different patch priors at various resolutions order to effectively reconstruct complete shape, enabling multi-resolution reasoning about the most informative learned patch priors to recover both global and local shape structures.
2 Related Work
2.1 3D Shape Reconstruction and Completion
Understanding how reconstruct 3D shapes an essential task for 3D machine perception. In particular, the task of shape completion to predict a complete shape from partial input observations has been studied by various works toward understanding 3D shape structures. Recently, many works have leveraged deep learning techniques to learn strong data-driven priors for shape completion, focusing on various different representations, e.g. volumetric grids [41, 13), continuous implicit functions [29, 21, 28, 7, 763, point clouds [30, 33], and meshes [11, 19]. These works tend to focus on strong synthetic supervision on a small set of train categories, achieving impressive performance on unseen shapes from train categories, but often struggling to generalize unseen classes. We focus

---

Mult-RResoluton Learning
Attention Map
Encoder
Input Encoding
Generated Encoding
Multi- Resolution Decoder
Input SDF
Completed SDF
Patch Prior Encodings
Overview of our approach. (a) shows our multi-resolution prior learning for shape completion. Each dotted block indicates patch prior learning for single resolution, and the three different resolution encodings are then fused in multi-resolution decoder that outputs a complete shape as an SDF grid. (b) illustrates our input partial observations and local input patches, from which we learn mappings to learned patch priors as to how to best compose the complete shape.
Figure 2:
on learning more generalizable, local shape priors in order to effectively reconstruct complete shapes on unseen class categories.
2.2 Few-Shot and Zero-Shot 3D Shape Reconstruction
Several works have been developed to tackle the challenging task few-shot or zero-shot shape reconstruction, observations in-the-wild often contain wide diversity of objects. In the few-shot scenario where several examples novel categories are available, Wallace and Hariharan [38] learn to refine given category prior. Michalkiewiez et al. [22] further propose to learn compositional shape priors for single-view reconstruction.
the zero-shot scenario without any examples seen for novel categories, Naeem al. [24] learn priors from categories to generate segmentation masks for unseen categories. Zhang et al. [44] additionally proposed to use spherical map representations to learn priors for the reconstruction of novel categories. Thai et al. [35] recently developed approach to transfer knowledge from an RGB image for shape reconstruction. We also tackle a zero-shot shape reconstruction task, by learning multi-resolution set of strong local shape priors to compose reconstructed shape.
Several recent works have explored learning shape priors by leveraging VQ-VAE backbone with autoregressive prediction to perform shape reconstruction [23, 42]. In contrast, we propose to learn multi-resolution shape priors without requiring any sequence interpretation, which enables direct applicability to real-world sean data that often contains noise and clutter. Additionally, hierarchical reconstruction has shown promising results for shape reconstruction [2, 6). Our approach also takes multi-resolution perspective, but learns explicit shape and local patch priors and their correlation to input partial observations for robust shape completion.
Method
3.1
Overview
Our method aims learn effective 3D shape priors that enable general shape completion from partial input scan data, on novel class categories not seen during training. Key to our approach is the observation that 3D shapes often share repeated local patterns for instance, chairs, tables, and nightstands all share a support surface, and chair or table legs can share similar structure with lamp rods. Inspired by this, we regard a complete object as a set of substructures, where each substructure represents local geometric region. We thus propose PatchComplete to learn such local priors and assemble them into a complete shape from a partial sean. An overview of our approach is shown in Figure 2.

---

Imput Encoder
Input Encoding
Input Local Features Q
Attention Maps
Input Partial Scan
Completed SDF P
Prior Encoder
Prior Encodings
Prior Local Features (K")
Chunk based patch resolution
Learnable shape priors (on training categories) (T*)
Chunked Prior Patches (V*)
Figure 3: Network architecture for local patch-based shape prior learning under a single resolution. We learn to build mappings from local regions partial object scan local priors based on complete learnable shape priors {T An input encoder takes the input of S to encode its local features (represented by patches Q). Analogously, we adopt a prior encoder to process each prior in {T to construct local prior feature pool {KC*). In parallel, we chunk priors in {T into patch volumes [v"). which are correspondingly fused by the attention to input patches to compose the most informative patch priors:
each incomplete local patch in Q to query the keys {K's from complete prior patches, and assemble their corresponding patch volumes {V*) for shape completion.
3.2 Learning Local Patch-Based Shape Priors
We first learn local shape priors from ground-trun -ruth train objects. We represent both the input partial scan S and the tround-truth shape G as 3D truncated signed distance field grids of size D Figure 3 illustrates the process of learning local shape priors. We learn to build mappings from local regions in incomplete input observations local priors based complete shapes, in order robustly reconstruct the complete shape output.
We first aim to learn patch-based priors, which extract from learnable global shape priors {Te). We denote G as the set of ground-truth train objects the e-th category. These {TT) priors are initialized per train category based on mean-shift clustering within shapes in G Thus T is set of representative samples in each category, which are encoded parallel to the input scan S.
Both encoders are analogously structured as 3D convolutional encoders which spatially downsample by factor of R, resulting are uniformly chunked into patches {Q:). i=1...., N3; similarly, the encoded priors (from (TT)) are chunked into patches {.
3D encoding of size N N D/R. The input encoding (from S)
We then use each local part Q; to query for the most informative encodings of complete local regions {KC)), J. building this mapping based on cross-altention [37]. Then for each input patch, we calculate its similarity with all local patches in representative shapes on training categories:
Attention( (C,nttionQQ,KK)) softmax(
d/2 ) (KG =1 N3 c= C)
(1)
where d is the dimension of the encoded vectors of Qi ICg; C' is the category number. We then reconstruct complete shape patches (P.) by:
P. = Attention(Q.. K) V
(2)
where V> = is the set of 3 chunks (of resolution R) from shapes all categories {T)), where each V is paired with !CS e IC. We can then recompose {P.} to the predicted full shape P.
Loss. We use an & reconstruction loss to train the learned local patch priors. Note that (T*) are learned along with the network weights, enabling learning the most effective global shape priors for the shape completion task.

---

M
Queries
[002)
Keys Values
D
Queries
M8
{0*}
Keys/ / Values
Completed
Input Partial Scan S
SDF O pred
Queries
Keys / Values
M
[CC]]
a
10')
Conv
Fixed
Figure 4: Network architecture for multi-resolution learning pipeline. We generate a complete shape from partial input scan S by fusing input local features and learned local priors in multi-scale fashion. We first extract the input local features {Q.) and the learned local priors {K*) J using the prior learning model (M R) in Section 3.2 under different resolutions (R=32, 8. 4). Then, we use
attention (Equation 3) to generate intermediate features {O"), which we then recursively fuse to decode a complete shape.
3.3 Multi-Resolution Patch Learning
In Section 3.2, we learn to complete shapes in single patch resolution R. Since local substructures may exist at various scales, we learn patch priors under varying resolutions, which we then use to decode complete shape. We use three patch resolutions (R=4, 8, 32), for which we learn patch priors. This results in three pairs of trained [input encoder, prior encoder) (see Figure 3). Given partial input scan S, each pair outputs set of 1) input patch features { {2 and 2) prior patch features KR,, under the resolution of R=4, 8, 32. In this section, we decode complete shape from these multi-resolution patch priors.
Since R stores the patch priors, we use it as the (key, value) term into attention module, where each input patch feature Off is used to query for the most informative prior features in KR under different resolutions, from which we complete partial patches in feature spaces with multi-scale fashion. We formulate this process by
O Concat [Q, Attention (C)) ] R 4, 3,
(3)
where we concatenate input patch feature with the attention result to compensate information loss on observable regions. This outputs {O,²]
} as the intermediate feature of the i-th patch; 1, 1..,. Na3 NR denotes the number of patches under the resolution R; NR = D/ R. We recompose all generated patch features into a volume R
{ 1, ..., 3g Note that the dimension of each grid (see Eq. I). Then O is with the dimension of N3 x 2d.
feature in @ and K equals
We then sequentially use 3D convolution decoders to upsample and concatenate O from low to high resolution to fuse all the shape features as in
O Rr+1 Concat [Upsamplo(OO**), OR.411 R 32, 8,4,
(4)
O Opred= = Conr(Upsanplee((k))
In Eq. 4, OR. has a lower resolution than R-11 We then use deconvolution layers to upsample RR match the resolution of R+++ and then concatenate them together. We recursively fuse RR. into RR++, which produces Ra as the final fusion result with the resolution of (D/4)3. An extra upsampling followed by convolution layer is adopted to upsample ORs into our shape prediction Opred with the dimension of 3
In training the feature fusion, we fix all parameters from the (input encoder, prior encoder) under the three resolutions, since they are pre-trained on Section 3.2 and provide better-leamed priors and attention maps under these strong constraints. The whole pipeline for this section can be found in Figure 4.
We use an É loss to supervise the TSDF value regression. We weight the & loss to penalize false predictions based on the predicted signs as Eq. 5 shows, which represents whether this voxel grid occupied or not. It is also used in the loss function in Section 3.2.
5

---

C Woec (O pred, get) + Wempty Corept pred Ogt) + & (Oprraect Ogt).
(5)
In Eq. 5, pred represents the false positive TSDF values, where the ground truth has negative signs and the prediction has positive signs, which in general indicates the missing predictions. pred represents false negative TSDF values, where the ground truth has positive signs and the prediction has negative signs, which indicates extra predictions. pred means those predicted TSDF values with the same signs the ground-truth. During training, choose the weight for false positive (Wempty) as 5, and the weight for false negative (Woer) as 3.
3.4 Implementation Details
We train our approach single NVIDIA A6000 GPU, using an Adam optimizer with batch size 32 for a synthetic dataset and batch size 16 for a real-world dataset, and an initial learning rate of 0.001. We train for 80 epochs until convergence, and then decrease the learning rate by half after epoch 50. We use the same settings for learning priors and the multi-resolution decoding, which train for 4 and 15.5 hours respectively. For additional network architecture details, refer to the supplemental material.
Note that for training for real scan data, we first pre-train on synthetic data and then fine-tune only the input encoder. We use 1 12 shape priors our method, which are clustered from 3202 train shapes, and represented as TSDFs with truncation of 2.5 voxels.
4 Experiments and Analysis
4.1 Experimental Setup
Datasets. We train and evaluate our approach on synthetic shape data from ShapeNet [5] as well as on challenging real-world scan data from ScanNet [10]. For ShapeNet data, we virtually scan the objects to create partial input scans, following [26, 13]. We use 18 categories during training, and test on 8 novel categories, resulting in 3,202/1,325 train/test models with 4 partial scans for each model.
For real data, we use real-scanned objects from ScanNet extracted by their bounding boxes, with corresponding complete target data given by Scan2CAD [1]. We use 8 categories for training, comprising 7,537 train samples, and test on 6 novel categories of 1,191 test samples.
For all experiments, objects are represented as 32 signed distance grids with truncation of 2.5 voxel units for ShapeNet and 3 voxel units for ScanNet. The objects in ShapeNet unit cube, while we keep the scaling for ScanNet objects, and save their voxel sizes separately to keep the real size information. Additionally, to train and evaluate on real data, all methods are first pre-trained on ShapeNet and then fine-tuned on ScanNet.
normalized into the
Baselines. We evaluate our approach against various state-of-the-art shape completion methods. We compare with state-of-the-as shape completion methods 3D-EPN [13] and IF-Nets [7]. which learn effective shape completion dense voxel grids and with implicit neural field representations, respectively, without any focus on unseen class categories. We further compare to the state-of-the-an few-shot shape reconstruction approach of Wallace and Hariharan [38] (referred to Few-Shot) leveraging global shape priors, which we apply in our zero-shot unseen category scenario. Finally, AutoSDF [23] uses VQ- module with transformer-base autoregressive model over latent patch priors to produce TSDF shape reconstruction.
Evaluation Metrics. To evaluate the quality of reconstructed shape geometry, we use L1 Chamfer Distance (CD) and Intersection over Union (IoU) between predicted and ground truth shapes. To evaluate methods that output occupancy grids, we use occupancy thresholds used by the respective methods to obtain voxel predictions, i.e. 0.4 for [38] and 0.5 for [7]. To evaluate methods that output signed distance fields, we extract the iso-surface at level zero with marching cubes [20]. 10K points
are sampled on surfaces for CD calculation. Both Chamfer distance and IoU are evaluated on objects in the canonical system, and report Chamfer Distance scaled by x x102.

---

Table 1: Quantitative comparison for shape completion on synthetic ShapeNet [5] data.
Chamfer Distance (x10²) /
IoU1
3D-
Few-
IF-
Auto- SDF [23]
Ours
3D-
Few-
IF-
Auto- SDF [23]
Ours
EPN
[13] Shot [38]Nets [7]
EPN
[13]Shot
[38
[7]
Bag
5,01
8,00
4.77
5.81
3.94
0.738
0.561
0,698
0.563
0.776
Lamp
8.07
15.10
5.70
6.57
4.68
0.472
0.254
0.508
0.391
0.564
Bathtub
4.21
7.05
4.72
5.17
3.78
0.579
0.457
0.550
0.410
0.663
Bed
5.84
10.03
5.34
6.01
4.49
0.584
0.396
0.607
0.446
0.668
Basket
7.90
8.72
4.44
6.70
5.15
0.540
0.406
0.502
0.398
0.610
Printer
5.15
9.26
5.83
7.52
4.63
0.736
0.567
0.705
0.499
0.776
Laptop
3.90
10.35
6.47
4.81
3.77
0.620
0.313
0.583
0.511
0.638
Bench
4.54
8.11
5.03
4.31
3.70
0,483
0.272
0.497
0.395
0.539
Inst-
5.48
9.75
5.37
5.76
4.23
0.582
0.386
0.574
0.446
0.644
Avg
=2e 1
=9e" 2
=le
=3e
+4e 2
19e 3
+le
14e"
16e
#le
Cat-
5.58
9.58
5.29
5.86
4.27
0.594
0.403
0.581
0.452
0.654
Avg
=2e
#1e-1
+5e
+5e
+8c
+le-3
#3e-1
+7e
+le-3
4.2
Evaluation on Synthetic Data
In Table I, we evaluate our approach in comparison with prior arts unseen class categories of synthetic ShapeNet [5] data. Our approach on learning attention-based correlation learned local shape priors results in notably improved reconstruction performance, with coherent global and local structures, as shown in Figure 5. Table 1, our work outperforms other baselines both instance-wise and category-wise. One of the key factors is that our method learns multi-scale patch information from seen categories to complete unseen categories with enough flexibility, while most of the other baselines are designed for 3D shape completion on known categories, which hardly leverage shape priors across categories.
Input Surface
3D-EPN [Dai 17] [Wallace et al. 19] [chibane et al. 20] [Mittal 22]
Few-Shat
IF-Nets
AutoSDF
Ours
Ground Truth
Figure 5: Qualitative comparison for shape completion synthetic ShapeNet [5] dataset.
4.3
Evaluation on Real Scan Data
Table evaluates our approach in comparison with prior arts on real scanned objects from unseen categories in ScanNet [10]. Here, input scans are not only partial but often contain noise and clutter; our multi-resolution learned priors enable more robust shape completion this challenging scenario. Results in Figure 6 further demonstrate that our approach presents more coherent shape completion than prior methods by using cross-attention with learnable priors, which better preserves the global structures in coarse and cluttered environments.
7

---

Table 2: Quantitative comparison with state of the art on real-world ScanNet [10] shape completion.
Chamfer Distance (x10²) /
IoU1
3D-
Few-
IF-
Auto- SDF [23]
Ours
3D-
Few-
IF-
Auto- SDF [23]
Ours
EPN
[13]Sho] [38]Nets [7]
EPN
[13]Shot
[38
Nets [7]
Bag
8.83
9.10
8.96
9.30
8.23
0.537
0.449
0,442
0.487
0.583
Lamp
14.27
11.88
10.16
11.17
9.42
0.207
0.196
0.249
0.244
0.284
Bathtub
7.56
7.77
7.19
7.84
6.77
0,410
0.382
0.395
0.366
0.480
Bed
7.76
9.07
8.24
7.91
7.24
0.478
0.349
0.449
0.380
0.484
Basket
7.74
8.02
6.74
7.54
6.60
0.365
0.343
0.427
0.361
0.455
Printer
8.36
8.30
8.28
9.66
6.84
0.630
0.622
0.607
0.499
0.705
Inst-
8.60
8.83
8.12
8.56
7.38
0.441
0.387
0.426
0.386
0.498
Avg
=2e
=2e
#7e"
=2e" 2
#6. 2
+2e
+le-
#3e
#le"
#9e 3
Cat-
9.09
9.02
8.26
8.90
7.52
0.440
0.386
0.426
0.389
0.495
Avg
=3e 1
=8e
=8e
=2e
+2e
33e
16e
=7e
#3e"
t5e
Input Surface
AutoSDF [Dai et [Wallace al. 19] [Chibane 20] [Mittal et 22]
3D-EPN
Few-Shot
IF-Nets
Ours
Ground Truth
Figure 6: Shape completion real-world ScanNet [10] object scans. Our method to learn mappings to multi-resolution learnable patch priors enables more coherent shape completion on novel categories.
Table 3: Ablation study on different patch resolutions. A multi-resolution approach gains benefits from both global and local reasoning.
ShapeNet [5]
ScanNet [10]
Inst- CD4
Cat-
Inst-
Cat-
Inst-
Cat-
Inst- IoUt
Cat-
CD}
IoUf
loUt
CD1
CDJ
IoUt
Ours (323 priors only)
11.97
11.62
0.35
0.37
10.40
11.33
0.41
0.39
+3e-2
+1e-2
+1e
+1e
+3e-2
+1e-1
+22-3
+55-3
Ours 38
priors only)
4.89
4.92
0.61
0.62
7.67
7.84
0.49
0.49
-
+33-2
+4.-4
-
+3c-2
+3e
+6e
+2e-3
Ours 3 priors only)
4.45
4.50
0.64
0.64
7.37
7.63
0.48
0.48
#le 2
-
+4e
+2c
+3e
+5e
+4e 3
+7e-3
Ours
4.23
4.27
0.64
0.65
7.38
7.52
0.50
0.50
-4e-2
+5c
+le-3
+le-3
166
-
-9e-3
3 +5e

---

4.4
Ablation Analysis
Does multi-resolution patch learning help shape completion for novel categories? In Table 3, we evaluate shape completion with each resolution comparison with our multi-resolution approach. Learning only global shape priors (i.e., 32 tends to overfit to seen train categories, while the local patch resolutions can provide more generalizable priors. Combining all results in complementary feature learning for the most effective shape completion results.
Does cross-attention to learn local priors help? We evaluate our approach to learn both local priors and their correlation input observations with rros-aattention in Table which shows that this enables more effective shape completion on unseen categories. Here, the no attention experiment replaces the attention score calculation Eq. by using MLPs (on concatenated input/prior features) to predict weights for each input-prior pair.
Table 4: Ablation study on attention used to learn input and patch prior correlations.
ShapeNet [5]
ScanNet [10]
Inst- CD|
Cat-
Inst- IoU1
Cat- IoU
Inst- CD4
Cat-
Inst- IoU
Cat-
CD4
CD$
IoUf
Ours (no attention)
4.90
4.98
0.61
0.62
7.80
8.09
0.49
0,48
Ours
4.69
4.74
0.61
0.63
7.58
7.84
0.48
0.49
Table 5: Effect of synthetic pre-training real- world ScanNet [10] object completion vs. training from scratch.
What is the effect synthetic pre-training for real scan completion? Table 5 shows the effect of synthetic pre-training for shape completion on real scanned objects. This en- courages learning more robust priors to output cleaner local structures as given in the synthetic data, resulting in improved performance on real scanned objects.
Inst- CD4
Cat- CD4
Inst- loUf
Cat-
loUf
Scratch Ours
7.61
7.73
0.50
0.50
7.38
7.52
0.50
0.50
Does learning the priors help comple- tion? In Table 6, we evaluate our learn- able priors in comparison with using fixed priors (by mean-shift clustering train objects) for shape completion on ShapeNet [5]. Learned priors receive gradient information to adapt to best re- construct the shapes, enabling improved performance over fixed set of priors.
Table 6: Ablation on learnable priors in comparison with fixed priors ShapeNet [5].
Inst- CDL
Cat- CD4
Inst- IoUf
Cat- IoUf
Ours (fixed priors)
4.31
4.34
0.64
0.65
Ours
4.23
4.27
0.64
0.65
What is the effect of the train category split? Our novel category split was designed based on the number of objects each category, to mimic real-world scenarios where object categories with larger numbers of observations are used for training.
To show that approach is independent of the category splitting strategy, we add two new settings for evaluation on ShapeNet [5]. We use the same overall categories (26) as originally, then randomly shuffle them for train/novel categories. Table 7 shows that all the splits achieve similar results, which indicates that our approach is robust across different splits. It because the learned patch priors share universal partial informa- tion among categories, which reduces the dependency on train categories for shape completion tasks. Figure shows gualitative results for novel category samples in the alternative splits.
Table 7: Ablation study on train category split on ShapeNet [5].
Inst- CD4
Cat- CDJ
Inst- IoUt
Cat-
IoUt
Category Split Category Split 2
4.30
4.29
0.65
0.66
4.19
4.25
0.68
0.67
Ours
4.23
4.27
0.64
0.65

---

Input Surface
Ours
Ground Truth
Input Surface
Ours
Ground Truth
Figure 7: Qualitative results for shape completion on ShapeNet [5] for novel category samples in alternative train category splits.
4.5 Limitations
While PatchComplcte has presented a promising step towards learning more generalizable shape priors, various limitations remain. For instance, output shape completion is limited by the dense voxel grid resolution representing fine-scale geometric details. Additionally, detected object bounding boxes are required for real scan data input to independent shape completion predictions, while formulation considering other objects in the scene or an end-to-end framework could learn more effectively from global scene contextual information.
4.6 Societal Impact
This work proposes a method for completion for unseen categories with less dependency on data capturing, with potential application to robotics or content creation scenarios. For instance, robot could potentially more efficiently grasp unseen objects when estimating complete shape structure; or self-driving cars could perform obstacle avoidance by robust 3D shape estimation for variety of objects. While potential real-world applications are promising, this requires careful consideration of personal data privacy, well any biases embedded in the training dataset.
5
Conclusion
We have proposed PatchComplete to learn effective local shape priors for shape completion that enables robust reconstruction for novel class categories test time. This enables learning shared local substructures across variety of shapes that can be mapped local incomplete observations by cross attention, with multi-resolution fusion producing coherent geometry at global and local scales. This enables robust shape completion even for unseen categories with different global structures, across synthetic as well as challenging real-world scanned objects with noise and clutter. We believe that such robust reconstruction of real scanned objects takes important step towards understanding 3D shapes, and hope this inspires future work understanding real-world shapes and scene structures.
Acknowledgements
This project funded by the Bavarian State Ministry of Science and the Arts and coordinated by the Bavarian Research Institute for Digital Transformation (bidt).
10

---

References
[1] Avetisyan, A., Dahnert, M., Dai, A., Savva, M., Chang, A.X., NieBner, M.: Scan2ead: Learning cad model alignment in rgb-d scans (2018)
[2] Bechtold, J., Tatarchenko, M., Fischer, V., Brox, T.: Fostering generalization in single-view 3d reconstruction by learning a hierarchy of local and global shape priors (2021) 3
[3] Chabra, R., Lenssen, J.E., Ilg. E., Schmidt, T., Straub, J., Lovegrove, S., Newcombe, R.: Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In: European Conference on Computer Vision. pp. 608-625. Springer (2020)
[4] Chang, A.X., Dai, A., Funkhouser, T., Halber, M., Niessner, M., Savva, M., Song, S., Zeng, A., Zhang, Y.: Matterport3D: Learning from RGB-D data in indoor environments. International Conference on 3D Vision (3DV) (2017)
[5] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M.. Song, S., Su, H., Xiao, J., Yi, L., Yu, F: Shapenet: An information-rich 3d model repository
(2015) 6, 9, 10,
[6] Chen, Z., Zhang, Y., Genova, K., Fanello, S., Bouaziz, S., Haene, C., Du, R., Keskin, C.. Funkhouser, T., Tang, D.: Multiresolution deep implicit functions for 3d shape representation
(2021)> 3
[7] Chibane, J., Alldieck, T., Pons-Moll, G.: Implicit functions in feature space for 3d shape
reconstruction and completion (2020) 2, 6, 8,8
[8] Choi, S., Zhou, Q.Y., Koltun, V.: Robust reconstruction of indoor scenes. In: 2015 IREE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5556-5565. IEEE (2015)
[9] Choy, C.B., Xu, D., Gwak, J., Chen, K., Savarese, S.: 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In: European conference computer vision. pp. 628-644. Springer (2016)
[10] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scannet: Richly- annotated 3d reconstructions of indoor scenes (2017) 1, 6, 9, 14
[11] Dai, A., Nießner, M.: Scan2mesh: From unstructured range to 3d meshes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5583 (2019) 1,2
[12] Dai, A., Nießner. M., Zollhöfer, M., Izadi, S., Theobalt. C.: Bundlefusson: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (TOG) 36(3), 24 (2017)*
[13] Dai, A., Qi. C.R., Nießner, M.: Shape completion using www.oee--rrddieeeeee enns and shape synthesis (2017) 1, 6, 7,8
[14] Deng, B., Lewis, J.P., Jeruzalski, T., Pons-Moll, G., Hinton, G., Norouzi, M., Tagliasacchi, A.: Nasa neural articulated shape approximation. In: European Conference Computer Vision. pp. 612-628, Springer (2020)
[15] Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3d object reconstruction from single image. In: Proceedings of the IEEE conference computer vision and pattern recognition. pp. 605-613 (2017)
[16] Genova, K., Cole, F., Sud, A., Sarna, A., Funkhouser, T.: Local deep implicit functions for 3d shape. In: Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4857-4866 (2020)
[17] Izadi, S., Kim, D., Hilliges,
D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al.: Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. Proceedings of the 24th annual ACM symposium on User interface software and technology. pp. 559-568. ACM (2011) 1)
O., Molyneaux,

---

[18] Jiang, C., Sud, A., Makadia, A., Huang, J.. Nießner. M., Funkhouser, T., et al.: Local implicit grid representations for 3d scenes. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6001-6010 (2020) 1
[19] Li, X., Liu, S., Kim, K., Mello, S.D., Jampani, V., Yang, M.H., Kautz, J.: Self-supervised single-view 3d reconstruction via semantic consistency. In: European Conference on Computer
Vision. pp. 677-693. Springer (2020) 2
[20] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics 21(4), 163-169 (1987) 6
[21] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space (2019) 1,2
[22] Michalkiewicz, M., Tsogkas, S., Parisot, S., Baktashmotlagh, M., Eriksson, A., Belilovsky, E.: Leamnnng compositional shape priors for few-shot 3d reconstruction (2021) 3
[23] Mittal, P., Cheng, Y.C., Singh, M., Tulsiani, S.: AutoSDF: Shape priors for 3d completion,
reconstruction and generation. In: CVPR (2022) 3, 6, 7,
[24] Naeem, M.F., Örnek, E.P., Xian, Y., Gool, L.V., Tombari, F.: 3d compositional zero-shot learning with decompositional consensus (2021) 3
[25] Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A.: Kinectfusion: Real-time dense surface mapping and tracking. In: Mixed and augmented reality (ISMAR), 2011 10th IEEE international symposium on. pp. 127-136. IEEE (2011)
[26] Nie, Y., Lin, Y., Han, X., Guo, S., Chang, J., Cui, S., Zhang, J., et al.: Skeleton-bridged point completion: From global inference to local adjustment. Advances in Neural Information Processing Systems 33, 16119-1613 (2020)
[27] Nießner, M., Zollhöfer, M., Izadi, S., Stamminger, M.: Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (TOG) (2013)
[28) Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: Deepsdf: Learning continuous signed distance funetions for shape representation (2019) 1,2
[29] Peng, S., Niemeyer, M., Mescheder, L., Pollefeys, M., Geiger, A.: Convolutional occupancy networks (2020)
[30] Stutz, D.,, Geiger, A.: Learning 3d shape completion under weak supervision. International Journal of Computer Vision 128(5), 1162-1181 (2020) 2
[31] Takikawa, T., Litalien, J., Yin, K., Kreis, K., Loop, C., Nowrouzezahraii, D., Jacobson, A., McGuire, M., Fidler, S.: Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11358- 11367 (2021)
1367
[32] Tang, J., Han, X., Pan, J., Jia, K.. Tong. X.: A brrieedd-bridged deep learning approach for generating meshes of complex topologies from single rgh images. In: Proceedings of the ieee/cvf conference on computer vision and pattern recognition. pp. 4541 44550 (2019)
[33] Tang, J., Lei, J., Xu, D., Ma, F,, Jia, K., Zhang, L.: Sa-convonet: Sign-agnostic optimization of convolutional occupancy networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6504-6513 (2021) 1,2
[34] Tatarchenko, M., Dosovitskiy, A., Brox, T.: Octree generating networks: Efficient convolu- tional architectures for high-resolution 3d outputs. In: Proceedings of the IEEE international conference on computer vision. pp. 2088-2096 (2017)
[35] Thai, A., Stojanov, S., Upadhya, V., Rehg, J.M.: reconstruction of novel object shapes from single images (2021)* 3
12

---

[36] Tretschk, E., Tewari, A., Golyanik, V., Zollhöfer, M., Stoll, C., Theobalt, C.: Patchnets: Patch-based generalizable deep implicit 3d shape representations (2021)
[37] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I: Attention is all you need. Advances in neural information processing systems 30
(2017) 4
[38] Wallace, B., Hariharan, B.: Few-shot generalization for single-image 3d reconstruction via priors (2019) 3, 6, 7,8
[39] Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, YGG.; Pixel2mesh: Generating 3d mesh models from single rgb images. In: Proceedings of the European conference on computer vision (ECCV). pp. 52-67 (2018)*
[40] Whelan, T., Leutenegger, S., Salas-Moreno, R.F. Glocker, B., Davison, A.J.: Flasticfusion: Dense slam without pose graph. Proc. Robotics: Science and Systems, Rome, Italy (2015)
[41] Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: A deep representation for volumetric shapes (2015)
[42] Yan, X., Lin, L., Mitra, N.J., Lischinski, D., Cohen-Or, D., Huang, H.: Shapeformer: Transformer-basee shape completion via sparse representation. arXiv preprint 1rX1v:2201.10326 (2022)* 3
[43] Yang, G., Huang, X., Hao, Z., Liu, M.Y., Belongie, S., Hariharan, B.: Pointflow: 3d point cloud generation with continuous normalizing flows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4541 4550 (2019)
[44] Zhang, X., Zhang, Z., Zhang, C., Tenenbaum, J.B., Freeman, W.T., Wu, J.: Learning to reconstruct shapes from unseen classes (2018) 3
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section XXX.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See abstract, Section 1 and Section
(b) Did you describe the limitations of your work? [Yes] Section 4.5
(c) Did you discuss any potential negative socictal impacts of your work? [Yes] Section 4.6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See the link in abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 3.4, Section 4.1, and the Data Generation section of the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Table 1, Table 2, and Table 3, more details can be found in the Category-wise Evaluations with Error Bars section of the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3.4.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets.... (a) If your work uses existing assets, did you cite the creators? [Yes] We use and cite ShapeNet [5] and ScanNet [10].
(b) Did you mention the license of the assets? [Yes] See the Data Generation section of the supplemental material.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 'r using/curating? [Yes] See the Data Generation section of the supplemental material. (c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] See the Data Generation section of the supplemental material.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
14

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "siWbwWdOdHg", "reviewer": "Reviewer_z3XK", "summary": "The paper tackles a challenging problem - the performance degradation on unseen categories in shape completion tasks. Inspired by the observation that different categories may share similar local geometric structures, the authors propose a method to learn multi-resolution patch priors and then reconstruct the complete shape using the recursively fused features. The results show a significant performance improvement in novel-category shape completion compared to baseline methods, and ablations have verified the importance of each component.", "questions": "1. I wonder about the criteria for splitting the datasets. In my opinion, the partition of the data has an enormous impact on performance (e.g., it is hard to complete a chair if we only have sofas as priors). If you randomly choose the training categories, I think it would be better to use cross-validation to show the impact of how the data is divided or the generalization performance. \n\n2. Most of the categories in the paper and supplementary only have simple geometry, such as bed or bathtub. Although you mention the lack of fine-scale geometric details in the results, I am still curious about performance on more complex data (such as the chair or bookshelf you mentioned in the video).\n\n3. What is the impact of the number of priors? In the ablation part, the authors show the results on different patch resolutions, and it seems only 4^3 priors can achieve competitive performance. Can I assume that local path priors play a major role, and it may have less diversity? So it may have redundancy, and the boundary of the number of priors is meaningful in real-world applications. Moreover, it would be better to show the memory usage for storing the priors.\n\n4. How about the domain independence of the proposed method? Have you tried the cross-dataset experiments? More specifically, can you train priors on ShapeNet, and test using ScanNet? Logically speaking, there should be a reasonable result.\n5. By the way, why choose TSDF as the input shape representation? Is it because it is easy to perform 3D convolution operations? Is it possible to change TSDF to point cloud?", "limitations": "Tthe paper bills itself that it can complete shapes for entirely unseen categories, but the authors do not illustrate the criteria of splitting the categories in the dataset, making the generalization of the proposed method on unseen categories obscure. It would be better to show more results on this to support the claim. \n\nAnother concern is the performance on categories with more complex geometries, logically, learning over the patch level could be helpful to improve the local geometric details in the synthesized results. Although the authors mention it in the limitations, the results of the method in its current form do not fully show the power of leveraging patch-level priors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Learning patch-level priors and the fusion pipeline of different resolution patch priors is moderately novel to me. The pipeline in this submission is technically sound. The submission is clearly written and well organized. The authors also provide the source code in supplementary material for better clarity of the techincal details. BTW, I’m a bit impressed that the completed shapes are mostly watertight and have continuous geometry. \n\nIn general, the evaluation part of the manuscript is good to me. The numerical results show a significant performance improvement over the baseline methods on both synthetic and real scan data. However, as the authors mentioned in limitations, visual and qualitative results on categories with fine geometric structures are not shown, which could have been a strong support to the merit of the proposed method. For the ablation study, the authors answer the importance of each designed component and show convincing numerical results of the effectiveness.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "sEC7eyVnfSj", "reviewer": "Reviewer_oDR1", "summary": "The submission introduces PatchComplete for 3D object-level shape completion from a partial input SDF voxel grid, with a focus on generalizing to unseen object categories (not just unseen instances). It evaluates on the synthetic ShapeNet and the real-world ScanNet, which contains 3D reconstructions of rooms (ScanNet annotations are used to segment out objects; pseudo-GT full object shapes are obtained via Scan2CAD). The paper contributes a scheme to learn patch-level shape priors. It also contributes a method that associates an input incomplete shape with that patch-level prior and then reconstructs the complete shape as a (learned) merging of that associated/weighted prior information. Comparisons are done to a diverse set of prior work and PatchComplete outperforms them, especially on unseen categories. A number of ablations are also included.", "questions": "Questions for the rebuttal:\n\n1) It is not clear to me whether the accompanying code will be released.\n\n2) I have several questions about Table 3:\na) The ablation of which resolutions to use in the multi-resolution scheme (Table 3) shows rather marginal improvements when going from only the finest resolution (4^3) to all three resolutions (Ours; 4^3, 8^3, 32^3). Are really both 8^3 AND 32^3 necessary? How does 4^3+8^3 and 4^3+32^3 perform? \nb) What are the standard deviations for these numbers? Are the marginal improvements significant? \nc) Especially on ScanNet, which is the more interesting dataset compared to ShapeNet for this task, shows results that are on par for 4^3 only and for Ours. I am not really convinced that this supports the claim that using multi-resolution leads to the \"most effective shape completion results\" (line 216).\n\n3) An ablation of the concatenation in Eq. 3 could be added, to get a sense of how much the patch prior and how much the input encoding can achieve on their own, and what their respective issues are. I.e., use only Q^R_i in one ablation and use only the Attention() in another ablation. This is not crucial though.\n\n4) What is the \"no attention\" ablation in Table 4? What is used instead of attention? Only Q^R_i directly, as I suggested in 3)? What else is changed?\n\n5) An overall ablation that uses fixed priors, uses no pre-training, uses no attention, and only uses resolution 4^3 would be interesting. How much do these tweaks contribute to the overall performance?\n\nSuggestions for improvement that are not relevant for the rebuttal:\n\n- I suggest to add qualitative results for the ablations, at least in the supplement. \n\n- How many shapes/priors does {T^c} contain? I.e. how many representative samples (in total) are there out of how many total shapes {G^c}?\n\n- Some related works on deep implicit local geometry representations could be added:\n\n* Genova et al. Local Deep Implicit Functions for 3D Shape, CVPR 2020\n* Charbra et al. Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction, ECCV 2020\n* Takikawa et al. Neural Geometric Level of Detail: Real-Time Rendering With Implicit 3D Shapes, CVPR 2021\n* Jiang et al. Local Implicit Grid Representations for 3D Scenes, CVPR 2020\n* Deng et al. NASA: Neural Articulated Shape Approximation, ECCV 2020\n\n- There's a superfluous period in line 95.", "limitations": "- Limitations are mentioned. \n\n- Societal impact is discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "_Positives_\n\nThe method is novel and enables good generalization to unseen categories, which is particularly useful for static 3D reconstruction of unseen environments with systems like KinectFusion.\n\nThe paper is written well in terms of language and clarity. \n\nThe experimental evaluation is done well. The ablations consider a multitude of factors of the method.\n\n_Negatives_\n\nI don't see much on the technical level that could potentially be used in other areas of CV/ML. The method consists of a core (attention into a learned patch prior) and a number of tweaks, which are evaluated in the ablations. The core part is a simple version of (non-sparse) dictionary learning. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JIYEIKdMl-5", "reviewer": "Reviewer_Bd3i", "summary": "This paper proposed a 3D Shape reconstruction method using local patch priors. The proposed method uses the multi-resolution patch priors based on the observation that within a 3D structure, some details are repetitive and often easier to be constructed first. The paper is well written and balanced ", "questions": "* the method is indeed effective over the test case, how about the efficiency? what is the average runtime compared to other signal-pass and one-resolution methods?", "limitations": "Natural limitations as the author also pointed out, the bounding boxes are required for the scan, which means this method is constrained to a predefined space and is not directly translatable to use in the wild. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ the presentation of the paper is well put together\n\n+ the illustration of the paper is detailed and easy to understand\n\n+ the idea of using local prior and multi-resolution is not necessarily brand new, but it is effective for the method the paper is proposing.\n\n+ based on the paper's experiment results, it indeed improved some real-world 3D reconstruction\n\n- it would be great if the author could showcase some reconstruction from the real world without a ground-truth scan. Just a static RGB photo and the 3D reconstructed object would help better demonstrate the strength of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GjpeeSdtSWg", "reviewer": "Reviewer_ngNb", "summary": "This paper proposes a deep network architecture for performing 3D shape completion on unseen shape categories by learning local geometric priors at multiple scales. By learning sets of features for distinct categories on patches sampled at different resolutions and fusing these features with those computed from an input partial scan from an unseen category, the network is able to leverage the local information  to achieve a better reconstruction. The method is validated qualitatively and quantitatively against state-of-the-art baselines, and a brief ablation study is provided to justify the multi-resolution aspect.", "questions": "Why is the softmax denominator in (1) $d/2$? This seems to depart from the usual $\\sqrt d$.\n\nHow are the partial shapes inputted to the pipelines as SDFs? Aren't the partial shapes generically not closed surfaces?\n\nWhy do the ground truth synthetic surfaces in Figure 5 have artifacts and imperfections?\n\nTo what extent is the method robust to some of the hyperparameters used? For example, in particular how important is the choice of resolutions and the number of layers in the pyramid in the multi-scale approach? Similarly, how are the coefficients for the loss terms chosen?\n\nHow is the clustering to initialize the shape priors performed? Is it done directly on the SDF representations?", "limitations": "Limitations are sufficiently discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper makes a nice step towards training shape reconstruction models that can handle out-of-distribution examples from unseen categories. The multi-scale approach is sensible, and the experimental results validate its effect.\n\nThe results that are shown are all quite low frequency, with the shapes having fairy limited local variation. I wonder why this method is unable to handle more detailed geometries. Is this due to the fact that such textures are not seen in the training categories, or is there a different bottleneck?\n\nThe method description is pretty heavy on notation and somewhat difficult to follow. It would be helpful to clearly define the different objects that are considered and clearly distinguish between geometry representations and latent high-dimensional encodings. To this end it's also important to fix subtle typos in the notation, e.g. $S$ should be $\\mathcal S$ on L107.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept