SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the template learning stage, our training split consists of objects that do not overlap with the 15 test scans of the DTU dataset. We provide the list of scans used for training in the supplementary. We evaluate our method on standard 15 scans used for evaluation by [34, 55, 54, 48, 36]. The resolution of each image in the dataset is  $1200 \times 1600$ . Unlike MonoSDF [58] which resizes the images to  $384 \times 384$  resolution for training, we input the actual image resolution to our pipeline. The other dataset, BlendedMVS, consists of 113 scenes captured from multiple views, where each image is of resolution  $576 \times 768$ . Unlike DTU, BlendedMVS dataset has not been employed before in the sparse view setting, and as such, specific view IDs to train and test are not available. We therefore manually select 3 views that have little overlap.
Baselines. For sparse views, we compare against the classical MVS method, COLMAP [42], and recent volume rendering-based methods such as NeuS [48], VolSDF [54] and MonoSDF [58]. We exclude SparseNeuS [25] evaluation in disparate settings because of its inability to reconstruct 3D geometry when the input images have significantly less overlap. For dense views, we additionally compare against the recent grid-based method, Voxurf [50]. A mesh from COLMAP point cloud output is obtained using Screened Poisson Surface Reconstruction [19].
Evaluation Metrics. Following standard evaluation protocol, we use Chamfer Distance (CD) between the ground truth point cloud and predicted meshes to quantify the reconstruction quality. On the DTU dataset, following prior works [54, 48, 58], we report CD scores. On the BlendedMVS dataset, we simply show qualitative results as is commonly done in the literature. In addition to this, for evaluation on the MobileBrick dataset, we use F1 score computed using the ground-truth mesh.
4.1 Results on DTU
Figure 1 and Table 1 show qualitative and quantitative results, respectively, in the presence of sparse view inputs (three, to be specific) on the DTU dataset. We observe from Table 1 that our method outperforms all other competing methods, including MonoSDF [58], on nine of the fifteen scenes in the test set, and achieves the lowest mean CD score (the next best adds an additional 0.09 CD score).
Figure 6: Dense view reconstruction results on the DTU dataset.
Table 1: Quantitative comparisons for sparse and disparate view 3D reconstruction on DTU dataset (Section 4.1). For Scan ID  ${110}, \times$  indicates that COLMAP could not reconstruct any points.
Chamfer Distance (CD) (↓)
Scan ID →	24	37	40	55	63	65	69	83	97	105	106	110	114	118	122	Mean
COLMAP[41]	4.45	4.67	2.51	1.9	2.81	2.92	2.12	2.05	2.93	2.05	2.01	×	1.1	2.72	1.64	2.56
VolSDF [54]	5.24	5.09	3.99	1.42	5.1	4.33	5.36	3.15	5.78	2.07	2.79	5.73	1.2	5.64	6.2	4.2
NeuS [48]	4.39	4.59	5.11	5.37	5.21	5.60	5.26	4.72	5.91	3.97	4.02	3.1	4.84	3.91	5.71	4.78
MonoSDF [58]	3.47	3.61	2.1	1.05	2.37	1.38	1.41	1.85	1.74	1.1	1.46	2.28	1.25	1.44	1.45	1.86
Ours	3.37	4.11	1.46	0.75	2.74	1.52	1.13	1.63	2.08	0.98	0.87	2.7	0.47	1.24	1.57	1.77
On the other six scans, our method is the second-best performing method behind MonoSDF. This is because our predicted templates are not always perfect for all objects, meaning, regions on the surface with no predicted templates potentially suffer from incomplete reconstructions. Other methods such as VolSDF [54] and NeuS [48] fail to even beat the classical MVS method, COLMAP [43], indicating that these methods either require dense view inputs[54] or require a significant amount of overlap on the input views [25] to faithfully reconstruct the surface. This emphasizes the effectiveness of our method on the disparate views case. Further, Figure 5 compares surface details on the reconstructed geometry against, MonoSDF [58], where it can be observed that our method produces sharper details, attributed mainly to the geometry guidance from learned template priors.
On dense view inputs (49-64), Figure 6 and Table 3 present qualitative and quantitative results, respectively, over the aforementioned techniques. We additionally compare against a recent method, Voxurf [50]. In this setting, all neural surface reconstruction
Table 2: Generalization of TPN on MobileBrick dataset.
F1 Score (↑)
Object →	Bridge	Camera	Colosseum	Castle
Ours (pre-trained TPN)	0.565	0.61	0.219	0.175
Ours (TPN re-training)	0.658	0.67	0.22	0.187
methods outperform COLMAP, indicating that the presence of dense view inputs is necessary to tap into the learning ability of neural rendering techniques for improved geometry reconstruction. We observe that our method outperforms the rest on five scans, which is higher than any other method (Voxurf-three scans, NeuS-four, MonoSDF-two, and VolSDF-one). Overall, our method achieves the second-best mean CD score, closely trailing behind Voxurf (a difference of 0.03 CD score). This is expected as feature-grid-based approaches tend to reconstruct sharper details compared to MLP [58]. But otherwise, we beat all other existing works owing to our neural template regularization scheme. More qualitative results on dense view inputs are provided in the Supplementary material.
4.2 Results on BlendedMVS
We also run our pipeline on a more challenging, also real-world, BlendedMVS dataset [53]. We use the sparse input setting and show comparisons against the same set of prior works, i.e., NeuS[48] and MonoSDF [58]. Figure 7 shows qualitative results on 4 object-scenes with complex geometries. Clearly, our method obtains the best surface reconstruction. NeuS is not able to reconstruct the
Table 3: Quantitative comparisons for dense-view 3D reconstruction on DTU dataset (Section 4.1).
Chamfer Distance (CD) (↓)
Scan ID →	24	37	40	55	63	65	69	83	97	105	106	110	114	118	122	Mean
COLMAP[41]	0.81	2.05	0.73	1.22	1.79	1.58	1.02	3.05	1.4	2.05	1.0	1.32	0.49	0.78	1.17	1.36
NeuS[48]	1.0	1.37	0.93	0.43	1.1	0.65	0.57	1.48	1.09	0.83	0.52	1.2	0.35	0.49	0.54	0.84
VolSDF [54]	1.14	1.26	0.81	0.49	1.25	0.7	0.72	1.29	1.18	0.7	0.66	1.08	0.42	0.61	0.55	0.86
MonoSDF [58]	0.83	1.61	0.65	0.47	0.92	0.87	0.87	1.3	1.25	0.68	0.65	0.96	0.41	0.62	0.58	0.84
Voxurf [50]	0.91	0.73	0.45	0.34	0.99	0.66	0.83	1.36	1.31	0.78	0.53	1.12	0.37	0.53	0.51	0.76
Ours	0.78	1.29	0.68	0.42	0.96	0.70	0.68	1.43	1.30	0.827	0.62	0.94	0.34	0.49	0.49	0.79
Figure 7: Qualitative reconstruction results for three disparate view inputs on the challenging BlendedMVS dataset [53]. See Section 4.2 and the Supplementary for a more details explanation.
geometry faithfully for any of the objects due to disparate view inputs. MonoSDF, on the other hand, despite using ground truth depth and normal cues, fails to reconstruct accurate geometry, especially for Clock and Sculpture objects. In contrast to this, our method can accurately reconstruct the geometric details owing to template guidance coming from Stage 1.
4.3 Generalization on MobileBrick Dataset
We test the generalization ability of our template prediction network (TPN) on a new dataset, MobileBrick [21], by comparing the reconstruction results when the neural templates were learned by a pre-trained TPN (on DTU) vs. when they were trained on MobileBrick. Note that overall, the models from these two datasets are quite different in terms of geometry and structure. The results in Table 2 and qualitative results in Figure 8a (a) show that the reconstruction qualities under the two scenarios are comparable, attesting to the generalizability of our TPN. As mentioned before, the metric used in this evaluation is F1 score as reported by MobileBrick dataset.
Figure 8: (a) Generalization of TPN on MobileBrick Dataset. (b) Effect on chamfer distance with # of Views (left) and Gaussians (right).
Reference Image
Figure 9: Effect of different optimization constraints (Figure 2, Section 3.2) employed in our pipeline.
Only SDF Constraint
Only Depth Constraint
SDF + Depth Constraint
In Figure 9, we demonstrate the need for employing different loss functions as described in Section 3.2 and depicted on the right side of Figure 2 and Table 4. We observe how the reconstructed geometry changes when  $\mathcal{L}_{sdf}$  and  $\mathcal{L}_{depth}$  are separately employed. Evidently, only SDF constraints cannot reconstruct surfaces due to the coarse supervision from template centers. On the other hand, depth regularization significantly improves the reconstruction but holes
Table 4: Impact of each constraint.
CD (↓)
Only SDF Constraint	2.70
Only Depth Constraint	1.96
SDF + Depth Constraint	1.52
still exist. Our full pipeline includes both the above constraints, resulting in improved reconstruction, as can be seen in the last column of Figure 9 and Table 4. We also study reconstruction quality as a function of the number of input views. Figure 8b depicts the behavior of two such methods i.e. NeuS and Ours, up to fifty views. Our method achieves the same Chamfer Distance (CD) score for twelve views as NeuS achieves for around thirty. This demonstrates the efficacy of our method to "catch" surface geometry even when sufficient view inputs are not present. In addition to this, we measure the effect on reconstruction quality by varying the number of Gaussians. As it can be seen in Figure 8b the surface reconstruction quality significantly improves as we increase the number of gaussians. The reconstruction quality saturates after 576 gaussians and hence we use 576 gaussians throughout our experiments.
5 Discussions, Limitations and Future Work
We present DiViNet for sparse multi-view 3D reconstruction. Our core idea is a simple two-stage framework with neural Gaussian templates learned across different scenes serving as surface priors to regularize the volume-rendering-based reconstruction. Experiments demonstrate quality reconstructions by DiViNet in both dense and sparse+disparate view settings, with clearly superior performance over state-of-the-art in the latter.
One current limitation of our method is its reconstruction speed. While comparable to NeuS [48], VolSDF [54], and MonoSDF [58], it is still slow due to the two-stage approach, especially when generalizing to a large number of real-world scenes. In addition to addressing these issues, e.g., dynamic templates, we would also like to replace the MLPs in DiViNet with more efficient grid-based techniques, e.g., instant NGP [32], and explore more general primitives such as convexes and super-quadrics in place of Gaussians.
We thank the anonymous reviewers for their valuable comments and Sherwin Bahmani and Yizhi Wang for discussions during rebuttal and helping with the renderings in the paper. This research is supported in part by a Discovery Grant from NSERC (No. 611370).
[1] M. Agrawal and L. S. Davis. A probabilistic framework for surface reconstruction from multiple images. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 2, pages II–II. IEEE, 2001.
[2] M. Atzmon and Y. Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2565-2574, 2020.
[3] M. Bleyer, C. Rhemann, and C. Rother. Patchmatch stereo-stereo matching with slanted support windows. In *Bmvc*, volume 11, pages 1-11, 2011.
[4] A. Broadhurst, T. W. Drummond, and R. Cipolla. A probabilistic framework for space carving. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001, volume 1, pages 388-393. IEEE, 2001.
[5] F. Darmon, B. Bascle, J.-C. Devaux, P. Monasse, and M. Aubry. Improving neural implicit surfaces geometry with patch warping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6260-6269, 2022.
[6] J. S. De Bonet and P. Viola. Poxels: Probabilistic voxelized volume reconstruction. In Proceedings of International Conference on Computer Vision (ICCV), volume 2, page 3, 1999.
[7] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882-12891, 2022.
[8] Q. Fu, Q. Xu, Y. S. Ong, and W. Tao. Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction. Advances in Neural Information Processing Systems, 35:3403-3416, 2022.
[9] Y. Furukawa, C. Hernández, et al. Multi-view stereo: A tutorial. Foundations and Trends® in Computer Graphics and Vision, 9(1-2):1-148, 2015.
[10] K. Genova, F. Cole, A. Sud, A. Sarna, and T. Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4857-4866, 2020.
[11] K. Genova, F. Cole, D. Vlasic, A. Sarna, W. T. Freeman, and T. Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7154-7164, 2019.
[12] M. Goesele, B. Curless, and S. M. Seitz. Multi-view stereo revisited. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), volume 2, pages 2402-2409. IEEE, 2006.
[13] A. Gropp, L. Yariv, N. Haim, M. Atzmon, and Y. Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10099, 2020.
[14] X. Gu, Z. Fan, S. Zhu, Z. Dai, F. Tan, and P. Tan. Cascade cost volume for high-resolution multiview stereo and stereo matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2495-2504, 2020.
[15] P.-H. Huang, K. Matzen, J. Kopf, N. Ahuja, and J.-B. Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2821-2830, 2018.
[52] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan. Mvsnet: Depth inference for unstructured multiview stereo. In Proceedings of the European conference on computer vision (ECCV), pages 767-783, 2018.
[53] Y. Yao, Z. Luo, S. Li, J. Zhang, Y. Ren, L. Zhou, T. Fang, and L. Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1790-1799, 2020.
[54] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
[55] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33:2492-2502, 2020.
[56] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. PixelNeRF: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[57] Z. Yu and S. Gao. Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1949-1958, 2020.
[58] Z. Yu, S. Peng, M. Niemeyer, T. Sattler, and A. Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint arXiv:2206.00665, 2022.
[59] S. Zagoruyko and N. Komodakis. Learning to compare image patches via convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4353-4361, 2015.
[60] J. Zhang, Y. Yao, S. Li, T. Fang, D. McKinnon, Y. Tsin, and L. Quan. Critical regularizations for neural surface reconstruction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6270-6279, 2022.
[61] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[62] X. Zhang, S. Bi, K. Sunkavalli, H. Su, and Z. Xu. NeRFusion: Fusing radiance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5449-5458, 2022.