SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\nabla_ {\mathbf {q}} \sum_ {i \in \mathcal {N} _ {\mathbf {q}}} g \left(\left\| \mathbf {q} - \mathbf {q} _ {i} \right\|\right) = \nabla_ {\mathbf {q}} \sum_ {i \in \mathcal {N} _ {\mathbf {q}}} \ln \left(\frac {1}{\left\| \mathbf {q} - \mathbf {q} _ {i} \right\|}\right) = \sum_ {i \in \mathcal {N} _ {\mathbf {q}}} \frac {\mathbf {q} - \mathbf {q} _ {i}}{\left\| \mathbf {q} - \mathbf {q} _ {i} \right\| ^ {2}}. \tag {6}
$$
To prevent excessively large forces, we apply gradient clamping during implementation. As illustrated in Figure 4(d)-bottom, the proposed Repulsive UDF effectively addresses the problem of UDF and achieves a more comprehensive surface reconstruction.
3.4 Optimization details
We supervise the coarse anchor locations  $\mathbf{X}_c$  using  $L_{1}$ -chamfer distance loss with respect to the ground truth points sampled sparsely using farthest point sampling (FPS). The UDF predictions are supervised by the UDF between query and ground truth points using  $L_{1}$  loss with clamping of maximal distances [34]. Following MCC [4], the colors are supervised by the colors of closest ground truth points within 0.1 radius from the query points with cross-entropy loss for each color channel. We randomly sample 550 query points from the 3D world space per training batch.
We conduct extensive experiments to show the representational power and generalization capability of NU-MCC for object-level single-view reconstruction using CO3D-v2 dataset [5]. Additionally, we show the zero-shot generalization for in-the-wild reconstruction of RGB-D iPhone capture, AI generated images [10], and ImageNet [11] in Section 4.2.
Dataset. We follow the task established from MCC [4] which uses CO3D-v2 dataset [5] for single-view object reconstruction. CO3D-v2 contains approximately 37,000 short videos of 51 object categories in real-world settings. We use the training-validation split from MCC all categories experiment. The reconstruction task focuses on the foreground objects identified by the segmentation masks provided in CO3D dataset. CO3D provides the point cloud of the object and depth maps obtained from the videos via COLMAP [6, 7], albeit noisy. The sparsely sampled point clouds (20,000 points) from the COLMAP reconstruction serve as ground truths. The objects are normalized to have zero-mean and unit-variance. The query points are sampled from  $[-3, 3]$  along each axis.
Metrics. To evaluate the geometry, we report the F1-score with distance threshold of 0.1 following MCC [4]. Additionally, we also show the  $L_{1}$ -chamfer distance (CD) for evaluations omitting the sensitivity of the chosen distance threshold in F1-score. We also introduce  $L_{1}$ -RGB distance metrics to evaluate the colors. The  $L_{1}$ -RGB distance calculates the  $L_{1}$ -norms of the RGB difference between the predicted points and their nearest ground truths located within 0.1 radius. We detail the mathematical formulation of the  $L_{1}$ -RGB distance in the supplementary material.
Training details. Our model is trained with an effective batch size of 512 using  $4 \times$  NVIDIA A100 GPUs for 100 epochs. One epoch takes approximately 2 hours. We follow the optimizer and 3D data augmentation of MCC. Adam optimizer [47] with base learning rate of  $10^{-4}$ , cosine schedule, and linear warm-up for the first  $5\%$  of iterations are used. 3D data augmentation is performed by random scaling of  $s \in [0.8, 1.2]$  and rotation  $\theta \in [-180^{\circ}, 180^{\circ}]$  along each axis.
Implementation details. We employ 200 coarse anchor representations. Each query point attends to its 4-nearest coarse anchor features and 4-nearest fine features in the feature aggregation. During training, we set the distance clamping for UDF supervision to 0.5. At test time, we shift points with an initial UDF prediction lower than 0.23 for 10 iterations. To calculate the repulsive force, we process batches of 48,000 points where each point considers  $k = 16$  nearest neighbors. We clamp the shifting gradient to  $[-0.03, 0.03]$  in each direction.
4.1 Results on CO3D-v2 Validation Set
The quantitative results on CO3D-v2 [5] validation set are summarized in Table 1. We first conduct an ablation study of our decoder design with occupancy as the output representation. The occupancy is supervised following MCC [4] where a query point is considered "occupied" if it is located within a radius of 0.1 to a ground truth point, and "unoccupied" otherwise. Utilizing our Neighborhood decoder with only coarse anchor features results in comparable metrics to MCC [4], but with  $15 \times$  inference speedup. The incorporation of fine features in the feature aggregation process produces notably more accurate colors and slightly better geometry.
Replacing the occupancy with UDF potentially enhances the geometry and color. However, as seen in Figure 5, iteratively shifting points towards surface results in holes due to some points clumping in
ARCHITECTURE	REPRESENTATION	L1-CD↓	F1↑	L1-RGB↓	RUNTIME (s)↓
MCC [4]	Occ	0.284	76.41	0.376	18.5
Ours (no fine)	Occ	0.292	76.8	0.374	1.2
Ours	Occ	0.282	79.0	0.340	1.5
Ours	UDF	0.264	78.2	0.316	3.2
RepUDF	0.237	83.8	0.320	3.5
Table 1: Quantitative results on CO3D-v2 [5] validation set. Results are obtained using 216k query points and averaged over three different views.
Figure 5: The effect of repulsive force. Left: Standard UDF. Right: Repulsive UDF.
Figure 6: Qualitative comparison on CO3D-v2 [5] validation set. Our method captures higher details on the seen part and predicts the occlusion more accurately.
corners and edges. The use of the repulsive field mitigates the hole artifacts and creates surface with uniformly distributed points, which overall produces the best geometry and color.
Figure 6 shows the qualitative comparison with MCC [4]. While MCC [4] fails to reconstruct the fine details, such as the eyes of the teddybear and the intricate structure of the motorcycle, our method recovers significantly better details on the seen part, while more sensibly reconstructing the occlusion.
4.2 Zero-Shot Generalization In-the-Wild
We show the generalization capability of NU-MCC for in-the-wild reconstructions of RGB-D iPhone capture, AI generated images (Stable Diffusion [10]), and ImageNet [11]. Off-the-shelf segmentation tool [48] and depth estimator [49] are used to generate the object masks and depth images of the ImageNet [11] and AI generated [10] images. These experiments are challenging due to the large distribution shifts of depth images, object categories, and scene settings. Figure 7 shows NU-MCC is capable of accomplishing faithful reconstructions in zero-shot settings, including challenging object classes such as lawnmower in ImageNet [11] example.
We present NU-MCC, a novel single-view 3D reconstruction model leveraging large-scale training. Our approach includes two key innovations: a Neighborhood decoder and Repulsive UDF that in combination result in fast inference speed and high-quality textured reconstruction. We show that our model is generalizable to challenging zero-shot settings with large distribution shifts from training data, such as iPhone capture, AI generated images, and images curated from the web.
<sup>1</sup>We discovered and rectified a bug in MCC [4] F1-score calculation. Additionally, we use a score threshold of 0.4 that results in the best chamfer distance. Thus, our reported F1-score is much higher than in MCC paper.
Figure 7: Zero-shot generalization in-the-wild. NU-MCC trained on CO3D-v2 [4] is capable of faithful reconstructions of challenging in-the-wild images with large distribution shifts of depths, object categories, and scene settings with higher details than MCC [4].
Limitation. While NU-MCC produces remarkable results compared to the state-of-the-art [4], it is still inherently challenging to reconstruct the occluded parts with high-fidelity details. Additionally, our results degrade when there is a significant amount of outliers and noise in the input depth, which may result in "spikes" or "bumps", although these artifacts can be alleviated by selecting a higher number of features in the feature aggregation.
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-RP-2021-024), and the Tier 2 grant MOE-T2EP20120-0011 from the Singapore Ministry of Education. The authors acknowledge the computational resources provided by the HPC platform of Xi'an Jiaotong University.
[1] L. Schmid, M. N. Cheema, V. Reijgwart, R. Siegwart, F. Tombari, and C. Cadena, “Incremental 3d scene completion for safe and efficient exploration mapping and planning,” arXiv preprint arXiv:2208.08307, 2022.
[2] R. Menon, T. Zaenker, and M. Bennewitz, "Viewpoint planning based on shape completion for fruit mapping and reconstruction," arXiv preprint arXiv:2209.15376, 2022.
[3] R. G. Boboc, E. Băutu, F. Gîrbacia, N. Popovici, and D.-M. Popovici, “Augmented reality in cultural heritage: An overview of the last decade of applications,” Applied Sciences, 2022.
[4] C.-Y. Wu, J. Johnson, J. Malik, C. Feichtenhofer, and G. Gkioxari, "Multiview compressive coding for 3d reconstruction," in CVPR, 2023.
[26] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su et al., "Shapenet: An information-rich 3d model repository," arXiv preprint arXiv:1512.03012, 2015.
[27] X. Sun, J. Wu, X. Zhang, Z. Zhang, C. Zhang, T. Xue, J. B. Tenenbaum, and W. T. Freeman, "Pix3d: Dataset and methods for single-image 3d shape modeling," in CVPR, 2018.
[28] X. Yu, Y. Rao, Z. Wang, Z. Liu, J. Lu, and J. Zhou, "Pointr: Diverse point cloud completion with geometry-aware transformers," in ICCV, 2021.
[29] P. Mittal, Y.-C. Cheng, M. Singh, and S. Tulsiani, "Autosdf: Shape priors for 3d completion, reconstruction and generation," in CVPR, 2022.
[30] X. Yan, L. Lin, N. J. Mitra, D. Lischinski, D. Cohen-Or, and H. Huang, "Shapeformer: Transformer-based shape completion via sparse representation," in CVPR, 2022.
[31] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar, “Neural fields in visual computing and beyond,” in Computer Graphics Forum, 2022.
[32] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, "Occupancy networks: Learning 3d reconstruction in function space," in CVPR, 2019.
[33] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in CVPR, 2019.
[34] J. Chibane, G. Pons-Moll et al., “Neural unsigned distance fields for implicit function learning,” in NeurIPS, 2020.
[35] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” in ECCV, 2020.
[36] X. Long, C. Lin, L. Liu, Y. Liu, P. Wang, C. Theobalt, T. Komura, and W. Wang, “Neuraludf: Learning unsigned distance fields for multi-view reconstruction of surfaces with arbitrary topologies,” in CVPR, 2023.
[37] Y.-T. Liu, L. Wang, W. Chen, X. Meng, B. Yang, L. Gao et al., "Neudf: Leaning neural unsigned distance fields with volume rendering," in CVPR, 2023.
[38] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger, "Convolutional occupancy networks," in ECCV, 2020.
[39] S. Lionar, D. Emtsev, D. Svilarkovic, and S. Peng, "Dynamic plane convolutional occupancy networks," in WACV, 2021.
[40] J. Chibane, T. Alldieck, and G. Pons-Moll, "Implicit functions in feature space for 3d shape reconstruction and completion," in CVPR, 2020.
[41] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, “Tensorf: Tensorial radiance fields,” in ECCV, 2022.
[42] C. Reiser, S. Peng, Y. Liao, and A. Geiger, "Kilonerf: Speeding up neural radiance fields with thousands of tiny mls," in ICCV, 2021.
[43] K. Genova, F. Cole, D. Vlasic, A. Sarna, W. T. Freeman, and T. Funkhouser, “Learning shape templates with structured implicit functions,” in ICCV, 2019.
[44] K. Genova, F. Cole, A. Sud, A. Sarna, and T. Funkhouser, "Local deep implicit functions for 3d shape," in CVPR, 2020.
[45] S. Giebenhain and B. Goldlücke, "Air-nets: An attention-based framework for locally conditioned implicit representations," in 3DV, 2021.
[46] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun, “Point transformer,” in ICCV, 2021.
[47] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in ICLR, 2015.
[48] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., "Segment anything," arXiv preprint arXiv:2304.02643, 2023.
[49] R. Ranftl, A. Bochkovskiy, and V. Koltun, “Vision transformers for dense prediction,” in ICCV, 2021.