SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 The training procedure of our method.
Input: Point cloud  $\mathbf{P}$ , pretrained occupancy network  $(\Psi, \Phi)$ , learning rate  $\alpha$   
Output: optimal kernel hyperparameters  $\theta^{*}$  and Nyström samples  $\mathbf{Z}^{*}$ $\widetilde{\mathbf{P}} =$  upsample  $(\mathbf{P})$ $\mathbf{X} = \Phi([\mathbf{P}, \widetilde{\mathbf{P}}])$ $\mathbf{Y} = [\mathbf{0}, \Psi \circ \Phi(\widetilde{\mathbf{P}})]$   
initialize  $\mathbf{Z}$  as  $m$  random features from  $\mathbf{X}$ , initialize  $\theta$   
for  $N$  times do  
    compute  $\hat{g}_{\theta, \mathbf{Z}}$  from  $\mathbf{X}, \mathbf{Y}$  (Equations 8 and 9)  
    compute  $\mathcal{L}^{\mathrm{tune}}$  for  $\hat{g}_{\theta, \mathbf{Z}}, \mathbf{X}$  and  $\mathbf{Y}$  (Equations 10 and 11)  
     $(\theta, \mathbf{Z}) \gets (\theta, \mathbf{Z}) - \alpha \nabla_{\theta, \mathbf{Z}} \mathcal{L}^{\mathrm{tune}}$   
end for
In this section, we evaluate our method using real and synthetic data both quantitatively and qualitatively. As our method builds on Poco [7], we compare our results to this baseline. We also compare to other state-of-the-art convolutional occupancy networks, such as Conv [57] and SAP [56]. We compare to the strategy based on test-time finetuning of a convolutional occupancy network (Conv [57]) in SAC [69]. Additionally, we compare to the strategy based on fitting an MLP to the point cloud in NP [44]. We compare also to classical learning-free Poisson reconstruction (SPSR [33]) where input point normals are estimated with PCA. For state-of-the-art convolutional occupancy networks Poco, Conv and SAP, we use their official publicly available pretrained models as trained on the same training split of ShapeNet [12] with 3k sized point clouds and noise of variance 0.005. We use the publicly available official implementations of SAP and NP. We present also an ablative analysis of the components of our method in Section 4.7. We provide additional results in the supplementary material. We note that from a computational stand point, for a 10k sized input point cloud, our method takes roughly 10 seconds to converge on a NVIDIA RTX A6000 GPU. The other test-time tuning based alternatives we compare to here, namely SAC and NP take roughly 1 and 5 minutes respectively.
We run all the experiments for 100 epochs using Adam optimizer with learning rate 0.1. We use the Falcon Library to implement the tuning of Nyström KRR in feature space. Regarding the input size and feature dimensions, We experimented with  $N_P = 10k$  (also 500 and 3k in ablation) and  $N_F = 32$  and  $m = 500$  inducing points.
Following seminal work, we evaluate our method and the competition w.r.t. the ground truth using standard metrics for the 3D reconstruction task. Namely, the volumetric Intersection over Union (IoU) when ground truth meshes are available, the L1 Chamfer Distance  $\mathbf{CD}_1$  ( $\times 10^2$ ) and euclidean distance based F-Score (FS) when ground truth points are available, and finally Normal Consistency (NC) when ground truth normals are available. We detail the expressions of these metrics in the supplementary material.
ShapeNet [12] consists of various instances of 13 different synthetic 3D object classes. Following [80], we use the first 20 shapes from the test split of each class for evaluation. We generate inputs of different sizes (10k, 3k, 500) while adding gaussian noise of standard deviation 0.005 following the literature (e.g. [7, 57]). ScanNet v2[18] is a challenging real dataset containing 1513 room scans as captured with an RBG-D camera. We use 100 scans from the test split for evaluation. We sample 10k points from the scans as input. Faust [6] consists of real scans of 10 human body identities in 10 different poses. We use the entire 100 scans for evaluation. We sample 10k points from the scans as inputs. We use the provided mesh registrations to compute IoU.
4.4 Object Level Reconstruction
Table 1: IoU ↑ of ShapeNet reconstruction.
Conv	SAC	NP	Poco	Ours
Chair	0,87	0,88	0,67	0,90	0,91
Lamp	0,77	0,75	0,64	0,80	0,87
Table	0,89	0,79	0,71	0,89	0,91
Mean	0,84	0,81	0,67	0,86	0,90
Table 3: FS ↑ of ShapeNet reconstruction.
Conv	SAC	NP	Poco	Ours
Chair	0,95	0,98	0,87	0,98	0,99
Lamp	0,92	0,94	0,83	0,95	0,96
Table	0,98	0,92	0,80	0,98	0,99
Mean	0,95	0,95	0,83	0,97	0,98
Table 2:  ${\mathrm{{CD}}}_{1} \downarrow$  of ShapeNet reconstruction.
Conv	SAC	NP	Poco	Ours
Chair	0,41	0,39	0,56	0,42	0,34
Lamp	0,47	0,44	0,72	0,56	0,42
Table	0,35	0,56	0,62	0,38	0,30
Mean	0,41	0,46	0,63	0,45	0,35
Table 4: NC ↑ of ShapeNet reconstruction.
Conv	SAC	NP	Poco	Ours
Chair	0,95	0,95	0,88	0,95	0,96
Lamp	0,9	0,89	0,90	0,90	0,92
Table	0,96	0,93	0,78	0,96	0,97
Mean	0,94	0,92	0,85	0,94	0,95
We show here reconstruction results from 10k sized noisy point clouds of classes Chair, Table and Lamp of ShapeNet. Convolutional occupancy models were trained with 3k sized noisy inputs. We report a comparison on IoU in Table 1,  $\mathrm{CD}_1$  in Table 2, FS in Table 3, and NC in Table 4. Figures 2 and 4 shows qualitative comparisons. Our method improves consistently across all metrics over its baseline Poco, as well as the other strategies. Methods that benefit from deep data priors (Conv, SAC, Poco, Ours) outperform data prior free overfitting (NP). We note also that these results show that the finetuning strategy SAC is not stable and does not improve consistently on its deep convolutional occupancy network Conv. This is also confirmed in the visual results in Figure2, where the finetuning can lead to breakage in the reconstruction. Our method delivers more robust reconstruction generally with less holes and artifacts. As it can be seen in the same figure, it can particularly recover from failure cases of its baseline Poco. We believe that many of the failures of the baseline Poco are due to
Figure 2: Qualitative comparison of ShapeNet reconstructions.
its lack of generalization w.r.t. the input size. Figure 4 shows additional visual comparisons to SPSR. We provide a numerical comparison to this classical method in the supplementary material.
4.5 Generalization to Real Scene Scans
Figure 3: Qualitative comparison of ScanNet v2 reconstructions.
We evaluate here the generalization from synthetic object training to real scenes. We show reconstruction results from 10k points sampled from real scans of ScanNet v2 in Table 5. As meshes are not available we can not compute IoU here. Our method (Ours(Poco)) based on Poco outperforms the baseline and the competition, demonstrating superior generalization ability. Figure 3 shows a few
Table 5: ScanNet reconstruction.
CD1↓	NC↑	FS↑
SPSR	2.27	0.74	0.68
SAP	1,12	0,73	0,67
Conv	1,22	0,71	0,60
SAC	0,97	0,76	0,77
Poco	1,11	0,77	0,79
Ours (Poco)	0,94	0,79	0,80
Ours (Conv)	0,70	0,77	0,79
Table 6: Faust reconstruction.
IoU↑	CD1↓	NC↑	FS↑
SPSR	-	0.48	0.91	0.91
SAP	0,90	0,29	0,93	0,98
Conv	0,85	0,41	0,92	0,95
SAC	0,77	0,30	0,92	0,95
Poco	0,90	0,32	0,93	0,97
Ours	0,92	0,26	0,95	0,99
Figure 4: Qualitative comparison to Screened Poisson Surface Reconstruction (SPSR). Input / Ours / SPSR, in this order.
qualitative results. While we offer improved performances compared to the competition, we note that the quality of our reconstruction remains limited still is such a challenging setting. We also show that our method can be successfully applied to other existing occupancy networks, as combining our strategy with baseline Conv (Ours(Conv)) leads to a considerable improvement in its performance.
4.6 Generalization to Real Articulated Shape Scans
Figure 5: Qualitative comparison of Faust reconstructions.
We evaluate here generalization from synthetic object training to real non-rigid human shapes. We show reconstruction results from 10k points sampled from real scans in Faust. As summarized in Table 6, both convolutional networks SAP and Poco offer good initial performances. Notice how the finetuning based SAC can improve of its baseline (Conv)  $\mathrm{CD}_1$  score while conversely worsening its IoU score. We outperform both our baseline and competing methods. Figure 5 shows a visual comparison of the results, where we recover from some failures of our baseline (Poco), and show limitations of the finetuning strategy in SAC. Figure 4 shows additional visual comparisons to SPSR.
4.7 Ablative Analysis
We conduct further ablative studies of our proposed method on the Lamp class of ShapeNet, as we found its shapes to be challenging and displaying many fine and complex features. Table 7 shows the performance of both our method and our baseline network Poco [7] for inputs of size 500, 3k and 10k. We note that Poco has been trained on 3k sized inputs. Notice first how the
performance of Poco improves in  $\mathrm{CD}_1$  while deteriorating in IoU. This shows the performance of this model does not necessarily scale with input size. Our method on the other hand shows healthy consistent improvements over the baseline Poco, which also scale with input size. We notice also that the performance gap w.r.t. Poco is slightly tighter in the sparsest setup (500 points). We believe this is due to the inductive bias of kernel methods favouring smoothness, as such a prior can prove less effective for extremely sparse data.
Table 8 shows an ablation of the design choices of our method. Our performance drops as expected whether we disable the kernel hyperparameters  $\theta$  finetuning (w/o  $\theta$ ), the Nyström samples  $\mathbf{Z}$  finetuning (w/o  $\mathbf{Z}$ ). We note also the importance of learning the regression in feature space. We notice that reconstruction often fails when we attempt to solve the regression in euclidean space. Hence our w/o Feat. version uses a Fourier positional encoding to stabilize this baseline. However, this version still under performs compared to our final model. Disabling the tuning all together (w/o tune) results in a performance drop as well. Increasing the number of number of Nyström samples in this context improves the performance.
Table 8 provides also an ablation the number of Nyström samples  $m$  while tuning (Ours). We find that generally the higher the number of inducing features  $\mathbf{Z}$  the better the performance, as it was the case without hyper-parameter optimization (w/o tune). This is expected as the approximation error of the Nyström KRR decreases with more inducing points. Additionally, more learnable inducing points implies more representation power for the KRR function. Increasing  $m$  comes however with increased computation overhead due to both solving the Nyström KRR with a higher number of inducing features  $\mathbf{Z}$  and also learning these features. Increasing  $m$  beyond a certain limit can also lead to overfitting. As our aim in this work is an efficient approach to improve generalization, we find a relatively small  $m$  value (500) already sufficient to produce satisfactory shape predictions and offers a good performance/compute overhead trade-off.
Table 7: Ablation of input size.
IoU↑	CD1↓	NC↑	FS↑
Poco(500)	0,71	0,85	0,86	0,84
Ours (500)	0,73	0,80	0,85	0,85
Poco(3k)	0,85	0,60	0,91	0,96
Ours (3k)	0,86	0,52	0,92	0,96
Poco(10k)	0,80	0,56	0,90	0,95
Ours (10k)	0,87	0,46	0,92	0,96
Table 8: Ablation of our model.
IoU↑	CD1↓	NC↑	FS↑
w/o tune (m = 1k)	0.84	0.61	0.92	0.96
w/o tune (m = 5k)	0.87	0.55	0.92	0.97
w/o tune (m = 10k)	0.87	0.53	0.92	0.96
w/o Feat.	0.45	2.23	0.79	0.58
w/o θ	0.86	0.52	0.92	0.95
w/o Z	0.86	0.53	0.92	0.96
Ours (m = 500)	0.87	0.42	0.92	0.96
Ours (m = 1000)	0.88	0.41	0.92	0.97
Ours (m = 5000)	0.86	0.43	0.91	0.96
Finally, we note that we chose to use an equal number of input and augmented samples (Equation 5) empirically as it gave the best overall results. It is possible that for high levels of noise, a higher number of augmented samples could prove beneficial. However, higher levels of noise for the input also imply less accurate predictions from the occupancy network, meaning less reliable pseudo-labels for the augmented samples.
Extreme generalization from relatively sparse point cloud is still challenging for our method as well as the literature and we will seek further improvement in future work. Although it offers robustness, the inherent inductive bias in our method also favours smoothness, which can prove not ideal for representing the highest level of detail. Ablation of number of the NKRR Nyström samples  $m$  in Table 8 shows that the KRR can possibly suffer from overfitting for large values of  $m$ , hence setting this hyperparameter can be challenging. We provide visualizations and a short discussion of the behavior of our method under some baseline failures in the supplementary material.
We presented a method for shape reconstruction from unoriented point cloud that combines the data prior of a preexisting deep reconstruction network and an efficient non-parametric interpolant. The resulting combination offers improved generalization over the network baseline, network test-time finetuning, and several state-of-the-art methods. We hope we can inspire more work in the direction of fine-tuning and transfer learning from preexisting feedforward occupancy networks, especially since there are many competing strategies that could be explored. For instance, while we propose here to improve the boundary decision by learning the shape function, an alternative strategy could be to tune solely the features instead.