SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rui Peng $^{1,2}$  Xiaodong Gu $^{3}$  Luyang Tang $^{1}$  Shihe Shen $^{1}$  Fanqi Yu $^{1}$  Ronggang Wang $^{\boxtimes,1,2}$
$^{1}$ School of Electronic and Computer Engineering, Peking University
2Peng Cheng Laboratory 3Alibaba Group
ruipeng@stu.pku.edu.cn rgwang@pkusz.edu.cn
Combining the signed distance function (SDF) and differentiable volume rendering has emerged as a powerful paradigm for surface reconstruction from multi-view images without 3D supervision. However, current methods are impeded by requiring long-time per-scene optimizations and cannot generalize to new scenes. In this paper, we present GenS, an end-to-end generalizable neural surface reconstruction model. Unlike coordinate-based methods that train a separate network for each scene, we construct a generalized multi-scale volume to directly encode all scenes. Compared with existing solutions, our representation is more powerful, which can recover high-frequency details while maintaining global smoothness. Meanwhile, we introduce a multi-scale feature-metric consistency to impose the multi-view consistency in a more discriminative multi-scale feature space, which is robust to the failures of the photometric consistency. And the learnable feature can be self-enhanced to continuously improve the matching accuracy and mitigate aggregation ambiguity. Furthermore, we design a view contrast loss to force the model to be robust to those regions covered by few viewpoints through distilling the geometric prior from dense input to sparse input. Extensive experiments on popular benchmarks show that our model can generalize well to new scenes and outperform existing state-of-the-art methods even those employing ground-truth depth supervision. Code will be available at https://github.com/prstrive/GenS.
Surface reconstruction from multi-view images is a cornerstone task in computer vision with many applications in virtual reality, autonomous driving, robotics, etc. Typical solutions [15, 16, 7, 43, 58, 56, 10, 36] in the past were mostly based on a multi-step pipeline, which includes depth estimation, depth fusion and meshing. Although they have demonstrated their excellent performance, the procedure is cumbersome and inevitably introduces cumulative errors. While several early works [31, 61] used differentiable surface rendering to directly reconstruct surfaces, recent works [34, 49, 60], inspired by the huge success of neural radiance field (NeRF) [27] in synthesizing novel views, follow the volume rendering [24] to represent the 3D geometry with an occupancy field [25] or signed distance function (SDF) [35] and can achieve more impressive results.
The key idea of these approaches is to train a compact multi-layer perceptrons (MLPs) to predict the implicit representation (e.g., SDF) of each sampled point on camera rays. The density of volume rendering is then regarded as a function of this implicit representation, and alpha-composition of samples is performed to produce the corresponding pixel color and geometry information. However, existing methods are hampered by requiring a lengthy per-scene optimization procedure and cannot generalize to new scenes, which makes them infeasible for many application scenarios. A recent method [23] attempts to address these issues through conditioning the SDF-induced model with
MVSNeRF
SparseNeuS
Ours
Figure 1: Qualitative comparisons on DTU and BlendedMVS datasets with sparse inputs.
COLMAP
SparseNeuS (ft)
Ours (ft)
features extracted from sparse nearby views. Nevertheless, its accuracy is limited due to the smooth reconstruction, and the multi-stage process it relies on is prone to introducing cumulative errors. In this paper, we seek to establish an end-to-end generalizable model which can efficiently infer finer 3D structure. Compared with existing methods [49, 60], this generalization system faces more challenging problems. First, it's non-trivial to efficiently represent the scene. Previous methods [29, 23, 3, 50] either build a global volume or employ feature projections, but they have proven to be either lacking in detail or unsuitable for view independent surface reconstruction. Second, relying only on the rendering loss is difficult to reconstruct compact geometry, since the multi-view consistency is ignored. And we found that the ordinary photometric consistency also cannot effectively solve this problem for our generalizable model because of the existence of ambiguous areas such as low-texture and reflection. Last but not least, since generalizable models heavily rely on aggregation quality, how to infer smooth geometry when the input is sparse is a thorny issue.
To this end, we introduce GenS to tackle these challenges. The main ideas behind are as follows: 1) We first construct a generalized multi-scale volume to represent the scene, which preserves global smoothness through the low-resolution volumes and recovers geometric details from high-resolution volumes. Meanwhile, low-dimensional features make our model more lightweight compared to a single large-width volume. 2) We introduce the multi-scale feature-metric consistency, which enforces multi-view consistency in the multi-scale feature space, to replace the common photometric consistency. Compared with the original image space, learnable multi-scale features can provide more discriminative representation, and the feature space can be self-enhanced during the generalization training process to continuously improve the matching accuracy. 3) Inspired by the fact that the reconstruction with dense inputs is more accurate, we propose a view contrast loss to force the model to better perceive the geometry of regions visible by few viewpoints through teaching the reconstruction from sparse inputs with dense inputs.
To demonstrate the quantitative and qualitative effectiveness of GenS, we conduct extensive experiments on DTU [12] and BlendedMVS [59] datasets. Results show that our model can outperform existing state-of-the-art generalizable method [23], and even recent method [40] which adopts the ground-truth depth for supervision. Compared with the per-scene overfitting methods [49, 60, 61, 34, 23], we can also achieve comparable or superior results with dense inputs. Some comparisons are shown in Fig. 1. In summary, our main contributions are highlighted below: a) We present a powerful representation based on our generalized multi-scale volume, which can efficiently reconstruct smooth and detail surfaces from multi-view images. b) We introduce a more discriminative multi-scale feature-metric consistency to successfully constrain the geometry, which helps the generalization model converge to the optimum. c) We propose a view contrast loss to improve the geometric smoothness and accuracy when the visible viewpoint is limited. d) Our model can be trained end-to-end and achieve state-of-the-art reconstructions in both generic setting and per-scene optimization setting.
Figure 2: Illustration of GenS. We first extract multi-scale features through a FPN network. The generalized multi-scale volume is then reconstructed with the corresponding scale feature. We employ the same blending strategy as [50] to estimate the appearance of each point on a ray, and adopt the volume rendering to recover the color of a pixel. We design the multi-scale feature-metric consistency to constrain the geometry as shown in the top right. For convenience, we omit some losses that will be detailed later.
Classical multi-view reconstruction. Reconstructing 3D geometry from multi-view images is a longstanding problem in the field of 3D vision. Classical algorithms mainly adopt depth-based or voxel-based methodology to solve this problem. Multi-view stereo (MVS) is a typical class of depth-based methods, which takes stereo correspondence from multiple images as the main cue to reconstruct depth maps. While previous traditional MVS methods [1, 43, 7, 6, 56, 44] relied on the hand-crafted similarity metrics, many recent learning-based methods [58, 10, 48, 36] apply deep learning to achieve more discriminative matching. These methods go through complicated procedures to retrieve surface, including depth filtering, fusion and meshing [15, 2], and are prone to cumulative errors. On the other hand, voxel-based methods [45, 18, 11, 32] directly model objects in a volume, but they are restricted to memory, which is the common drawback of the volumetric representation, and cannot achieve high accuracy.
Neural surface. Due to the notable advantages of being able to achieve high spatial resolution, neural implicit functions have recently gained a lot of attention and have emerged as an effective representation of 3D geometry [47, 25, 35, 8, 26, 30, 37, 42] and appearance [20, 27, 21, 33, 46, 62, 38, 28]. Furthermore, some works [27, 22, 31] have proposed to train models without 3D supervision via differentiable rendering, e.g., surface rendering and volume rendering. Methods adopt surface rendering [31, 61, 65] only consider a single surface intersection point for each ray and fail to reconstruct complex objects, and they are restricted by the need of accurate object masks and careful weight initialization. On the contrary, recent methods use volume rendering [34, 49, 60, 64] to take multiple points along the ray into consideration and achieve more impressive results. However, either type of method requires an expensive per-scene optimization and cannot generalize to new scenes.
Generalizable neural surface. In the field of novel view synthesis, some methods [3, 50, 63, 13] have successfully introduced the generalization into rendering methods. These methods suffer from the same problem as NeRF: the geometry is ambiguous. Few works have focused on the generalization of neural surface reconstruction. A recent study, SparseNeuS [23], is the first attempt to achieve this by reconstructing the surface from nearby viewpoints in a multi-stage manner. Nevertheless, its reconstruction lacks details, and same to the classical 3D reconstruction, the multi-stage pipeline may accumulate errors at each stage. On the contrary, our designed model can be trained end-to-end and reconstruct smoother and more refined geometries.
Given  $N$  posed images of an object taken from different viewpoints, our goal is to reconstruct the surface as an implicit function without expensive per-scene optimization or only by fast fine-tuning. Our overall framework is depicted in Fig 2. We first introduce how to infer the geometry and appearance from the generalized multi-scale volume in Sec. 3.1, then elaborate on the necessity and implementation of the multi-scale feature-metric consistency in Sec. 3.2, and finally detail the realization of view contrast loss in Sec. 3.3 and the overall pipeline in Sec. 3.4.
3.1 Geometry and color reasoning from the generalized multi-scale volume
Compared with existing solution [23], which relies on a single volume and multi-stage strategy, we have three main advantages. First of all, our generalized multi-scale volume is a more powerful representation, which implicitly decouples geometry into base structures in low-resolution volumes and high-frequency details in high-resolution volumes. Second, with the low-dimensional features, we can construct multi-scale volumes with higher resolution and less memory consumption than a single large-width volume. Besides, our model can be trained end-to-end, avoiding cumulative errors.
Generalized multi-scale volume construction. Suppose there are  $N$  images  $\{I_i\}_{i=0}^{N-1}$  of an object, we first apply the FPN network [19] to extract multi-scale feature maps  $\{F_i^j\}_{i,j=0,0}$  for all images with shared weights, and different volumes are then constructed from features at corresponding scales. In this paper, we define a bounding box of interest in the reference frustum like [23] and in the world coordinate system like [49, 66] when dense inputs are available. We adopt a combination of  $L$  volumes  $\{V_j\}_{j=0}^{L-1}$ , which cover the same region but with different resolutions  $Ch \times \frac{D}{2^j} \times \frac{H}{2^j} \times \frac{W}{2^j}$ .
Here, we discuss at the first scale and omit the scale subscript  $j$  for convenience. Given camera intrinsics  $\{K_i\}_{i=0}^{N-1}$  and extrinsics  $\{[R,t]_i\}_{i=0}^{N-1}$ , we first project the voxel  $v = (x,y,z)$  onto viewpoint  $i$ 's pixel position:
$$
q _ {i} (v) = \pi \left(K _ {i} R _ {i} ^ {T} \left(v - t _ {i}\right)\right), \tag {1}
$$
where  $\pi((x, y, z)^T) = (\frac{x}{z}, \frac{y}{z})^T$  is an operator to convert homogeneous coordinates to cartesian coordinates. Then we can get the corresponding feature of each voxel on  $i_{th}$  viewpoint through bilinear interpolation  $F_i(v) = F_i < q_i(v) >$ . To fuse features from all viewpoints  $\{F_i(v)\}_{i=0}^{N-1}$ , we adopt the same aggregation strategies to generate cost volume as in [50] that concatenates mean and variance to simultaneously capture statistical and semantic information:  $B(v) = [Mean(v), Var(v)]$ .
Simply repeating the above process on features and volumes of all  $L$  scales, we can get the multi-scale cost volumes  $\{B_j\}_{j=0}^{L-1}$ . Next, we further design an efficient multi-scale 3D network  $\psi$  to refine these cost volumes in one forward, starting from the finest volume and injecting the others into different stages of the model to save memory. The output of the 3D network  $\{V_j\}_{j=0}^{L-1} = \psi(\{B_j\}_{j=0}^{L-1})$  is the multi-scale volume that we need to infer the geometry.
Geometry reasoning. For an arbitrary 3D point  $p = (x, y, z)$ , we first get the interpolation of volumes at all scales  $\{V_{j}(p)\}_{j=0}^{L-1}$  through trilinear sampling, and then concatenate them as the final feature  $\mathcal{F}(p) \in \mathbb{R}^{Ch_1}$ , where  $Ch_1 = L \times Ch$ . Combining the feature and the point position, an MLP network is applied to predict the corresponding SDF value:  $sdf_{\theta}: \mathbb{R}^3 \times \mathbb{R}^{Ch_1} \to \mathbb{R}$ . And the surface is represented by the zero-level set of the SDF value:
$$
S = \{p \in \mathbb {R} ^ {3} | s d f _ {\theta} (p, \mathcal {F} (p)) = 0 \}. \tag {2}
$$
Color prediction. We refer to the first viewpoint  $I_0$  as the reference image. To predict the color of each point on a ray, we employ the blending strategy similar to [50]. We first project the 3D point  $p$  to source views' pixel position according to Eq. 1, and interpolate the corresponding colors  $\{I_i(p)\}_{i=1}^{N-1}$  and features  $\{F_i(p)\}_{i=1}^{N-1}$ . Here, we only use the highest resolution features to predict the color. Next, an MLP network takes the concatenation of features and viewing direction differences  $\Delta d = d - d_i$  as input, to predict the softmax-activated blending weights  $\{w_i(p)\}_{i=1}^{N-1}$  of each source view, and the final color is blended as the weighted sum of source colors:
$$
c (p) = \sum_ {i = 1} ^ {N - 1} I _ {i} (p) w _ {i} (p). \tag {3}
$$
SDF-based volume rendering. Given the density  $\{\sigma_i\}_{i=1}^M$  and color  $\{c_i\}_{i=1}^M$  of  $M$  samples along the ray  $p(t) = o + td$  emitting from camera center  $o$  to pixel  $q$  in view direction  $d$ , NeRF [27] approximates the color using numerical quadrature:
$$
\hat {C} = \sum_ {i = 1} ^ {M} T _ {i} \alpha_ {i} c _ {i}, T _ {i} = \prod_ {j = 1} ^ {i - 1} (1 - \alpha_ {j}), \tag {4}
$$
Figure 3: Multi-view aggregation ambiguity. Here, we take two viewpoints as an example. (a) For those low-texture regions, sampling points near the surface may get the same aggregation and lack discriminability. (b) The aggregation of points away from the surface are random and hard to infer the accurate geometry, e.g., two sampling points may get the same aggregation even with different SDF value.
where  $T_{i}$  is the accumulated transmittance, and  $\alpha_{i} = 1 - \exp (-\sigma_{i}\delta_{i})$  in original volume rendering. To better approximation the geometry of the scene, NeuS [49] proposed an unbiased and occlusion-aware weighting method to incorporate signed distance, and the  $\alpha_{i}$  is formulated as:
$$
\alpha_ {i} = \max  \left(\frac {\Phi_ {s} (s d f (p \left(t _ {i}\right))) - \Phi_ {s} (s d f (p \left(t _ {i + 1}\right)))}{\Phi_ {s} (s d f (p \left(t _ {i}\right)))}, 0\right). \tag {5}
$$
Here,  $\Phi_s(x) = (1 + e^{-sx})^{-1}$  is the sigmoid function and  $s$  is an anneal factor. Readers can refer to [49] for more details.
3.2 Multi-scale feature-metric consistency
Rendering loss tends to trap the model into sub-optimization since it only considers a single point and ignores the consistency among multiple viewpoints. To mitigate this problem, a straightforward practice is to project the image patches of multiple views to the estimated surface location based on the local planar assumption and rely on the photometric consistency to enforce the multi-view consistency. However, we found this solution works well for per-scene overfitting training [4, 5] but brings limited benefits to generalization training.
We analyze that the main reason may be the failure of photometric consistency, which becomes more challenging for generalization training. As proven in recent self-supervised multiview stereo methods [55, 54, 57, 39], the assumption of photometric consistency isn't always effective, and the predicted geometry still has significant holes even in combination with the robust patch similarity like SSIM [52]. As the coordinate-based methods train models separately for each scene to directly overfit the scene, they have greater potential to converge to the optimum. However, our generalization model encodes all scenes with one model, and it requires image features to infer geometry, which makes the model rely heavily on the discriminability of features, e.g., regions like low-texture and reflection become more critical for degrading results. As shown in Fig 3 (a), those regions violating photometric consistency not only reduce
Figure 4: Multi-scale feature space. The feature space is more discriminative than ordinary image space, and is more potential to find the corresponding point during matching.
the accuracy of multi-view matching, but also decrease the discriminability of generalization model's input (we call this aggregation ambiguity), while the input of overfitting methods are distinct (3D coordinate).
To overcome these challenges, we propose the multi-scale feature-metric consistency to measure the consistency between views in a multi-scale feature space, as shown in Fig. 4. There are three main advantages of doing this way. First of all, the learnable feature is proven to be more discriminative than the original image [14], especially on those ambiguous regions like low-texture and reflection. Second, due to the larger receptive field, multi-scale information is conducive to improving the matching accuracy, and allows the model to be assisted by global information while recovering details. More importantly, the feature discriminability can be continuously self-enhanced in the process of generalization training. The multi-scale feature space can train a powerful model through more accurate matching, and the more powerful model can in turn lead to a more discriminative feature space. And the enhanced feature can further mitigate the aforementioned aggregation ambiguity. These advantages have been proven in Tab. 3.
To generate the geometry, we adopt the same approximate method as [5] to directly locate the zero-level set of the SDF. As shown in Fig. 5, We first find the interval where a ray intersects the surface by checking whether the signs of the SDFs of two adjacent sampling points are different. To handle occlusion, we only extract the surface within the first interval. Suppose the two samples of the interval are  $p_1$  and  $p_2$ , and their distances to the camera center are  $t_1$  and  $t_2$  respectively, our goal is to compute the position of  $p_s$ . Here, we rely on an assumption that two adjacent samples are close enough that the near surface can be regarded as a local plane. In this way, we can get two similar triangles:
$$
\triangle p _ {2} p _ {s} s _ {2} \sim \triangle p _ {1} p _ {s} s _ {1}, \tag {6}
$$
Figure 5: Locating the surface of a ray.
Therefore, we can approximate the distance from the surface to the camera center  $t_{s}$  as:
$$
\frac {- s d f \left(p _ {2}\right)}{s d f \left(p _ {1}\right)} = \frac {t _ {2} - t _ {s}}{t _ {s} - t _ {1}} \Rightarrow t _ {s} = \frac {s d f \left(p _ {1}\right) t _ {2} - s d f \left(p _ {2}\right) t _ {1}}{t _ {1} - t _ {2}}. \tag {7}
$$
We thus can get the coordinate of the surface point  $p_{s} = o + t_{s}d$ .
Through the automatic differentiation of the SDF network at  $p_s$ , we can get the corresponding normal  $n_s$ . Based on the assumption that the local surface centered at  $p_s$  is a plane of normal  $n_s$ , we can find the corresponding pixel position  $q_i$  in  $i_{th}$  source view that correspond to the pixel  $q_0$  in reference view:
$$
q _ {i} = H _ {i} q _ {0}, H _ {i} = K _ {i} \left(R _ {i} R _ {0} ^ {T} + \frac {R _ {i} \left(R _ {i} ^ {T} t _ {i} - R _ {0} ^ {T} t _ {0}\right) n _ {s} ^ {T}}{n _ {s} ^ {T} p _ {s}}\right) K _ {0} ^ {- 1}. \tag {8}
$$
For a pixel patch  $\mathbf{q}_0$  in the reference view, we can find the corresponding source patch through passing all pixels to Eq. 8 like  $\mathbf{q}_i = H_i\mathbf{q}_0$ . Regardless of occlusion, if the estimated surface  $p_s$  is accurate, then these corresponding patches should also be consistent. In this paper, we measure patch consistency in a multi-scale feature space. We only apply features at the top 3 scales, since features at lower scales lose a lot of structural information. Therefore, for a pixel patch at a certain view, we can get the multi-scale patches  $\{F_j < \mathbf{q} >\}_{j = 0}^2$  through bilinear interpolation, and we upsample and concatenate them together as input  $F'$ , whose channel is  $Ch_2 = 3\times Ch$ , for patch similarity measure. Here, we employ the normalization cross correlation (NCC) to compute the feature-space consistency:
$$
N C C _ {i} = \frac {1}{C h _ {2}} \sum_ {l = 0} ^ {C h _ {2} - 1} \frac {\operatorname {C o v} \left(F _ {0 l} ^ {\prime} , F _ {i l} ^ {\prime}\right)}{\sqrt {\operatorname {V a r} \left(F _ {0 l} ^ {\prime}\right) \operatorname {V a r} \left(F _ {i l} ^ {\prime}\right)}}, \tag {9}
$$
where  $Cov$  denotes covariance and  $Var$  refers to variance. Following the common solution in multi-view stereo field [7], we compute the final multi-scale feature-space consistency loss as the average of the best  $K$  NCCs:
$$
L _ {m f c} = \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \left(1 - N C C _ {k}\right). \tag {10}
$$
3.3 View contrast loss
For a 3D structure captured by multiple viewpoints, there is a fact that some regions are covered by enough viewpoints, while some regions are only visible to a few viewpoints. Compared with the
former, the aggregated features of the latter are more likely to be polluted by irrelevant rays, making them less predictable. To solve this problem, we design a view contrast loss to improve the accuracy of the reconstruction when visible views are limited, which enforces the geometric estimation to be the same under different inputs of the same scene.
We empirically let results from dense inputs to supervise results of sparse inputs. Specially, taking a set of multi-view images as input, we first reconstruct a multi-scale volume as a teacher, which is used to infer the finer SDF value  $s$  for a set of 3D points  $P$ . Then we build a student multi-scale volume from sparse input views and estimate the corresponding SDF value  $s'$ . Meanwhile, as shown in Fig. 3 (b), we found that only the sampling points falling on the surface have positive epipolar correspondences, and their aggregated features are more meaningful, while other samples are more random, and may obtain the same aggregation even if their SDF values are different. As shown in Fig. 6, we thus only calculate the consistency loss for near-surface points, whose finer SDF values are more accurate:
Figure 6: Visualization of view contrast loss.
$$
L _ {v c} = \frac {1}{\left| P ^ {\prime} \right|} \sum_ {p \in P ^ {\prime}} | s (p) - s ^ {\prime} (p) |, \tag {11}
$$
where  $P^{\prime}$  is a set of points close to the surface inferred from the fine SDF according to Eq. 7.