SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3.4 Overall pipeline
This section will introduce some implementation details and crucial components of our model including generalization training and fine-tuning.
Loss function. The overall loss function is defined as:
$$
L = L _ {\text {c o l o r}} + \alpha L _ {m f c} + \beta L _ {v c} + L _ {\text {r e g}}. \tag {12}
$$
For a batch of sampled pixel set  $Q$ , the color loss is computed as the L1 distance between the rendered color and the ground-truth:
$$
L _ {\text {c o l o r}} = \frac {1}{| Q |} \sum_ {q \in Q} | C (q) - \hat {C} (q) |. \tag {13}
$$
To make the geometry more compact and accurate, we apply the regularization loss which is composed of four terms:
$$
L _ {r e g} = \gamma L _ {e k} + \eta L _ {\text {s m o o t h}} + \lambda L _ {t v} + \delta L _ {\text {s p a r s e}}. \tag {14}
$$
Eikonal loss [9] is employed to regularize SDF values of all sampled points  $P$ :
$$
L _ {e k} = \frac {1}{| P |} \sum_ {p \in P} \left(\left\| \nabla s d f (p) \right\| _ {2} - 1\right) ^ {2}. \tag {15}
$$
To maintain the smooth of the surface, we introduce a regularization to the gradient of the normal:
$$
L _ {\text {s m o o t h}} = \frac {1}{| Q |} \sum_ {q \in Q} \| n _ {\text {g r a d}} (q) \| _ {2}, \tag {16}
$$
where  $n_{\text{grad}}(q)$  is the alpha composition of normal gradient  $\nabla^2 \text{sd} f(q)$  in a ray through pixel  $q$ . Besides, we also adopt the total variation (TV) regularization [41] for our multi-scale volumes:
$$
L _ {t v} = \sum_ {j = 0} ^ {L - 1} \sqrt {\Delta_ {x} ^ {2} \left(V _ {j}\right) + \Delta_ {y} ^ {2} \left(V _ {j}\right) + \Delta_ {z} ^ {2} \left(V _ {j}\right)}. \tag {17}
$$
To clean the geometric estimation, we introduce a sparsity prior:
$$
L _ {s p a r s e} = \frac {1}{| P |} \sum_ {p \in P} \exp (- \tau | s d f (p) |). \tag {18}
$$
Method	24	37	40	55	63	65	69	83	97	105	106	110	114	118	122	Mean
VolRecon* [40]	1.20	2.59	1.56	1.08	1.43	1.92	1.11	1.48	1.42	1.05	1.19	1.38	0.74	1.23	1.27	1.38
PixelNerf [63]	5.13	8.07	5.85	4.40	7.11	4.64	5.68	6.76	9.05	6.11	3.95	5.92	6.26	6.89	6.93	6.28
IBRNet [50]	2.29	3.70	2.66	1.83	3.02	2.83	1.77	2.28	2.73	1.96	1.87	2.13	1.58	2.05	2.09	2.32
MVSNerf [3]	1.96	3.27	2.54	1.93	2.57	2.71	1.82	1.72	2.29	1.75	1.72	1.47	1.29	2.09	2.26	2.09
SparseNeuS [23]	1.68	3.06	2.25	1.10	2.37	2.18	1.28	1.47	1.80	1.23	1.19	1.17	0.75	1.56	1.55	1.64
GenS	1.45	2.77	1.69	0.97	1.54	1.90	1.03	1.49	1.36	0.97	1.07	0.97	0.62	1.14	1.16	1.34
NeuS [49]	4.57	4.49	3.97	4.32	4.63	1.95	4.68	3.83	4.15	2.50	1.52	6.47	1.26	5.57	6.11	4.00
VolSDF [60]	4.03	4.21	6.12	0.91	8.24	1.73	2.74	1.82	5.14	3.09	2.08	4.81	0.60	3.51	2.18	3.41
IBRNet (ft)	1.67	2.97	2.26	1.56	2.52	2.30	1.50	2.05	2.02	1.73	1.66	1.63	1.17	1.84	1.61	1.90
COLMAP [44]	0.90	2.89	1.63	1.08	2.18	1.94	1.61	1.30	2.34	1.28	1.10	1.42	0.76	1.17	1.14	1.52
SparseNeuS (ft)	1.29	2.27	1.57	0.88	1.61	1.86	1.06	1.27	1.42	1.07	0.99	0.87	0.54	1.15	1.18	1.27
GenS (ft)	0.91	2.33	1.46	0.75	1.02	1.58	0.74	1.16	1.05	0.77	0.88	0.56	0.49	0.78	0.93	1.03
Table 1: Quantitative results of Chamfer Distance on DTU dataset with sparse inputs. \* denotes that the method needs the ground-truth depth for supervision.
Generalization training. We select  $N = 4$  for sparse setting and  $N = 19$  for dense setting. We use Adam optimizer [17] with the base learning rate of 1e-3 for feature network and 5e-4 for other MLPs. We train the joint loss for 16 epochs on two A100 GPUs. We increase the value of  $\alpha$  from 0 to 1 and in the first 2 epochs. In our implementation, we generate the surface points  $P'$  of each image of the model trained with dense input first, and then distill the model with sparse input, with  $\beta$  set to 1. We build the generalized multi-scale volume with 5 scales, whose resolution increases from  $2^4$  to  $2^8$ . Each volume is equipped with thin features with only 4 feature channels, which allows us to save memory compared to general single volume methods.
Fine-tuning. After generalization training, we first reconstruct the generalized multi-scale volume, which has encoded the geometry information. Then we sparse the multi-scale volume by pruning voxels far from the surface. During fine-tuning, we abandon the feature network, and directly optimize the multi-scale volume and MLPs. With the generalization prior, we can achieve state-of-the-art performance in only about 20 minutes of fine-tuning.
We demonstrate the state-of-the-art performance of GenS with comprehensive experiments and verify the effectiveness of each module through ablation studies. We first introduce the datasets and then analyze our results.
Datasets. We conduct experiments on both DTU [12] and BlendedMVS [59] datasets as previous methods [49, 60, 23]. Our generalization model is trained on DTU dataset, which is an indoor MVS dataset with 124 different scenes scanned from 49 or 64 views with fixed camera trajectories. Following [61, 49, 23], we take the same 15 scenes for testing. The training set is defined as in [58, 36], and the test scenes contained therein are removed. We also evaluate our model on BlendedMVS, which is a large-scale synthetic dataset. Each scene is scanned from different number of views, and all images has a resolution of  $768 \times 576$ . We report the Chamfer Distance for DTU, and show some visual effects for BlendedMVS.
4.1 Comparisons
Results on DTU. We first adopt the same testing split and configuration as [23] to compare with existing generalizable methods [3, 50, 23, 63]. The results shown in Tab. 1 indicate that our model outperforms existing methods by a significant margin, and this advantage can be amplified after rapid fine-tuning (about 20 mins). Even compared with recent method [40], which adopts the ground-truth depth for supervision, our model can achieve superior results. The qualitative results in Fig. 1 show that our reconstruction exhibits finer details. We further conducted more experiments on DTU to compare with per-scene overfitting methods with more input views. The quantitative comparisons in Tab. 2 show that our model can surpass some methods [61, 34, 49, 60] just through a very fast network inference, i.e., we can achieve more than  $34\%$  improvement on scene 24 compared with [49]. After a fast fine-tuning, the performance can be significantly improved, and even surpassing recent SOTA works [4, 5, 51]. Some visualization results in Fig. 7 depict that our model trained on large amounts of data is more robust to ambiguous regions.
Method	24	37	40	55	63	65	69	83	97	105	106	110	114	118	122	Mean
IDR [61]	1.63	1.87	0.63	0.48	1.04	0.79	0.77	1.33	1.16	0.76	0.67	0.90	0.42	0.51	0.53	0.90
MVSDF [65]	0.83	1.76	0.88	0.44	1.11	0.90	0.75	1.26	1.02	1.35	0.87	0.84	0.34	0.47	0.46	0.88
COLMAP [44]	0.45	0.91	0.37	0.37	0.90	1.00	0.54	1.22	1.08	0.64	0.48	0.59	0.32	0.45	0.43	0.65
NeRF [27]	1.90	1.60	1.85	0.58	2.28	1.27	1.47	1.67	2.05	1.07	0.88	2.53	1.06	1.15	0.96	1.49
UNISURF [34]	1.32	1.36	1.72	0.44	1.35	0.79	0.80	1.49	1.37	0.89	0.59	1.47	0.46	0.59	0.62	1.02
VolSDF [60]	1.14	1.26	0.81	0.49	1.25	0.70	0.72	1.29	1.18	0.70	0.66	1.08	0.42	0.61	0.55	0.86
NeuS [49]	1.00	1.37	0.93	0.43	1.10	0.65	0.57	1.48	1.09	0.83	0.52	1.20	0.35	0.49	0.54	0.84
HF-NeuS [51]	0.76	1.32	0.70	0.39	1.06	0.63	0.63	1.15	1.12	0.80	0.52	1.22	0.33	0.49	0.50	0.77
Voxurf [53]	0.65	0.74	0.39	0.35	0.96	0.64	0.85	1.58	1.01	0.68	0.60	1.11	0.37	0.45	0.47	0.72
NeuralWarp [4]	0.49	0.71	0.38	0.38	0.79	0.81	0.82	1.20	1.06	0.68	0.66	0.74	0.41	0.63	0.51	0.68
Geo-NeuS* [5]	0.46	0.83	0.38	0.39	0.88	0.61	0.51	1.26	0.92	0.68	0.57	0.82	0.30	0.41	0.42	0.63
GenS	0.66	1.01	0.71	0.43	1.06	0.99	0.73	1.43	1.18	0.78	0.64	0.93	0.38	0.54	0.54	0.80
GenS (ft)	0.55	0.71	0.39	0.38	0.79	0.65	0.57	1.29	0.96	0.64	0.49	0.59	0.33	0.44	0.45	0.62
Figure 7: Qualitative comparisons with per-scene overfitting methods on DTU dataset with dense inputs.
Generalizing to BlendedMVS. Like [23], we conduct additional verification on BlendedMVS to showcase the generalization ability of our model. We also employ the same evaluation strategy as [23] for a fair comparison. As shown in Fig. 1, just through a fast network inference, our model can recover more geometric details than SparseNeuS [23]. Through the fine-tuning of 5k iterations, the effect will be significantly improved and better than SparseNeuS with 10k iterations.
Ablation studies. We conduct ablation studies on DTU dataset to understand how the components of our model contribute to the overall performance. We start with our baseline model SparseNeuS [23], and gradually insert our contributions. The results in Tab. 4 show that our full model combining all components has the best mean score, and the baseline model, without any of our contributions, performs the worst. Multi-scale Feature-metric Consistency (MFC): Our self-enhanced MFC can continuously improve the multi-view consistency of the model, and we also elaborated the ablation results of its three main characteristics in Tab. 3. And the baseline is based on the pixel-wise feature consistency proposed in [65]. It can be
Table 2: Quantitative results of Chamfer Distance on DTU dataset with dense inputs. \* denotes that we retrain the Geo-NeuS without sparse geometric supervision.
Patch-similarity	Multi-scale	Self-enhanced	Mean
X	X	X	1.86
✓	X	X	1.76
✓	✓	X	1.73
✓	✓	✓	1.62
Table 3: Some ablation studies on MFC.
Figure 8: Reconstruction from volumes with different resolution.
seen that our strategy is more robust and efficiency. Generalized Multi-scale Volume (GMV):
MFC	GMV	VCL	24	37	40	55	63	65	69	83	97	105	106	110	114	118	122	Mean
✘	✘	✘	2.26	3.39	2.04	1.27	2.47	2.65	1.62	1.84	1.61	1.32	1.82	1.94	0.91	1.78	1.62	1.90
✓	✘	✘	1.61	3.12	1.99	1.16	2.00	2.21	1.30	1.58	1.45	1.18	1.48	1.53	0.80	1.54	1.43	1.62
✓	✓	✘	1.51	3.07	1.88	0.97	1.56	2.11	1.12	1.45	1.31	0.95	1.20	1.02	0.64	1.32	1.24	1.42
✓	✓	✓	1.45	2.77	1.69	0.97	1.54	1.90	1.03	1.49	1.36	0.97	1.07	0.97	0.62	1.14	1.16	1.34
Table 4: Ablation results on DTU.
Figure 9: Visualization of some ablation results on DTU.
We show some results of the models have different resolutions in Fig. 8. We can see that the reconstruction of a single high-resolution volume is unbearably noisy (higher-resolution volume will lead to more empty voxels, which is more tricky for generalizable models.) and overly smooth at low resolution, whereas our GMV reconstructs clean and detailed geometry. And our representation is lighter than a single volume with a resolution of  $32 \times 192^3$  due to our thin feature. View Contrast Loss (VCL): As the results shown in Tab. 1 and Tab. 2, the reconstruction with dense inputs is more accurate than the sparse reconstruction, we therefore treat the former as a teacher to teach the latter. The results shown in Tab. 4 validate that this strategy can indeed improve the reconstruction quality of the model. The visualization of the ablation results are shown in Fig. 9, which further depicts that every contributions we propose can continuously improve performance.
**Limitation.** Although our model exhibits excellent generalization performance in multi-view reconstruction, we found that it cannot satisfactorily handle scenes with large camera motion, such as surrounding cases. Because in these scenarios, the aggregation features will be polluted by the ray features shooting from behind. Our current solution is to first predict the local structure covered by some adjacent viewpoints like [23, 58, 36], and finally fuse them together.
In this paper, we introduced GenS, an end-to-end generalizable neural surface reconstruction model. We first encode all scenes into our generalized multi-scale volume, a more powerful representation that can reconstruct clean and detailed 3D structures. Then we introduce the multi-scale feature-metric consistency to combat the challenge of the photometric consistency failure. The learnable multi-scale feature can provide more discriminative representation and can be self-enhanced during the generalization training. And we finally designed a view contrast loss to improve the accuracy of the reconstruction through distilling the finer reconstruction from dense inputs to the reconstruction from sparse inputs. Experimental results on both DTU and BlendedMVS datasets show that our model possess stronger generalization ability and can achieve start-of-the-art reconstruction through fast network inference or efficient fine-tuning. In the future, we will focus on improving the performance of the model in difficult scenarios.
This work is financially supported by National Natural Science Foundation of China U21B2012 and 62072013, Shenzhen Science and Technology Program-Shenzhen Cultivation of Excellent Scientific and Technological Innovation Talents project(Grant No. RCJC20200714114435057), Shenzhen Science and Technology Program-Shenzhen Hong Kong joint funding project (Grant No. SGDX20211123144400001), this work is also financially supported for Outstanding Talents Training Fund in Shenzhen. In addition, we thank our collaborators in Alibaba Group and the anonymous reviewers for their valuable comments.
[1] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM TOG, 28(3):24, 2009.
[2] Fausto Bernardini, Joshua Mittleman, Holly Rushmeier, Cláudio Silva, and Gabriel Taubin. The ball-pivoting algorithm for surface reconstruction. IEEE TVCG, 5(4):349-359, 1999.
[3] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, pages 14124-14133, 2021.
[4] François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. Improving neural implicit surfaces geometry with patch warping. In CVPR, pages 6260–6269, 2022.
[5] Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. Geo-neus: geometry-consistent neural implicit surfaces learning for multi-view reconstruction. NeurIPS, 2022.
[6] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE TPAMI, 32(8):1362-1376, 2009.
[7] Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In ICCV, pages 873-881, 2015.
[8] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In ICCV, pages 7154-7164, 2019.
[9] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10099, 2020.
[10] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In CVPR, pages 2495-2504, 2020.
[11] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In UIST, pages 559-568, 2011.
[12] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In CVPR, pages 406-413, 2014.
[13] Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In CVPR, pages 18365-18375, 2022.
[14] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711. Springer, 2016.
[15] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In SGP, volume 7, page 0, 2006.
[16] Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM ToG, 32(3):1-13, 2013.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[18] Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. IJCV, 38:199-218, 2000.
[19] Tsung-Yi Lin, Piotr Dólar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pages 2117-2125, 2017.
[20] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NeurIPS, 33:15651-15663, 2020.
[21] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In CVPR, pages 2019-2028, 2020.
[22] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3d supervision. NeurIPS, 32, 2019.
[23] Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In ECCV, pages 210-227. Springer, 2022.
[24] Nelson Max. Optical models for direct volume rendering. IEEE TVCG, 1(2):99-108, 1995.
[25] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, pages 4460-4470, 2019.
[26] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations as layers in neural networks. In ICCV, pages 4743-4752, 2019.
[27] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, pages 405-421, 2020.
[28] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM ToG, 41(4):1-15, 2022.
[29] Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-end 3d scene reconstruction from posed images. In ECCV, pages 414-431. Springer, 2020.
[30] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In ICCV, pages 5379-5389, 2019.
[31] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, pages 3504-3515, 2020.
[32] Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM ToG, 32(6):1-11, 2013.
[33] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In ICCV, pages 4531-4540, 2019.
[34] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV, pages 5589-5599, 2021.
[54] Hongbin Xu, Zhipeng Zhou, Yu Qiao, Wenxiong Kang, and Qixia Wu. Self-supervised multi-view stereo via effective co-segmentation and data-augmentation.
[55] Hongbin Xu, Zhipeng Zhou, Yali Wang, Wenxiong Kang, Baigui Sun, Hao Li, and Yu Qiao. Digging into uncertainty in self-supervised multi-view stereo. In ICCV, pages 6078-6087, 2021.
[56] Qingshan Xu and Wenbing Tao. Multi-scale geometric consistency guided multi-view stereo. In CVPR, pages 5483-5492, 2019.
[57] Jiayu Yang, Jose M Alvarez, and Miaomiao Liu. Self-supervised learning of depth inference for multi-view stereo. In CVPR, pages 7526-7534, 2021.
[58] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In ECCV, pages 767-783, 2018.
[59] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In CVPR, pages 1790–1799, 2020.
[60] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. NeurIPS, 34:4805-4815, 2021.
[61] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurIPS, 33:2492-2502, 2020.
[62] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In ICCV, pages 5752-5761, 2021.
[63] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR, pages 4578-4587, 2021.
[64] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. In NeurIPS, volume 35, pages 25018-25032, 2022.
[65] Jingyang Zhang, Yao Yao, and Long Quan. Learning signed distance field for multi-view surface reconstruction. In ICCV, pages 6525-6534, 2021.
[66] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.