Title: Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D Reconstruction

SDF

Neural Rendering

Implicit Representations

SLAM
Abstract: Learning neural implicit representations has achieved remarkable performance in 3D reconstruction from multi-view images. Current methods use volume rendering to render implicit representations into either RGB or depth images that are supervised by the multi-view ground truth. However, rendering a view each time suffers from incomplete depth at holes and unawareness of occluded structures from the depth supervision, which severely affects the accuracy of geometry inference via volume rendering. To resolve this issue, we propose to learn neural implicit representations from multi-view RGBD images through volume rendering with an attentive depth fusion prior. Our prior allows neural networks to sense coarse 3D structures from the Truncated Signed Distance Function (TSDF) fused from all available depth images for rendering. The TSDF enables accessing the missing depth at holes on one depth image and the occluded parts that are invisible from the current view. By introducing a novel attention mechanism, we allow neural networks to directly use the depth fusion prior with the inferred occupancy as the learned implicit function. Our attention mechanism works with either a one-time fused TSDF that represents a whole scene or an incrementally fused TSDF that represents a partial scene in the context of Simultaneous Localization and Mapping (SLAM). Our evaluations on widely used benchmarks including synthetic and real-world scans show our superiority over the latest neural implicit methods.

Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors

Abstract

Learning neural implicit representations has achieved remarkable performance in 3D reconstruction from multi-view images. Current methods use volume rendering to render implicit representations into either RGB or depth images that are supervised by multi-view ground truth. However, rendering a view each time suffers from incomplete depth at holes and unawareness of occluded structures from the depth supervision, which severely affects the accuracy of geometry inference via volume rendering. To resolve this issue, we propose to learn neural implicit representations from multi-view RGBD images through volume rendering with an attentive depth fusion prior. Our prior allows neural networks to perceive coarse 3D structures from the Truncated Signed Distance Function (TSDF) fused from all depth images available for rendering. The TSDF enables accessing the missing depth at holes on one depth image and the occluded parts that are invisible from the current view. By introducing a novel attention mechanism, we allow neural networks to directly use the depth fusion prior with the inferred occupancy as the learned implicit function. Our attention mechanism works with either a one-time fused TSDF that represents a whole scene or an incrementally fused TSDF that represents a partial scene in the context of Simultaneous Localization and Mapping (SLAM). Our evaluations on widely used benchmarks including synthetic and real-world scans show our superiority over the latest neural implicit methods. Please see our project page for code and data at https://machineperceptionlab.github.io/Attentive_DF_Prior/ .

1 Introduction

3D reconstruction from multi-view images has been studied for decades. Traditional methods like Structure from Motion (SfM) [61], Multi-View Stereo (MVS) [62], and Simultaneous Localization and Mapping (SLAM) [49] estimate 3D structures as point clouds by maximizing multi-view color consistency. Current methods [84, 28] mainly adopt data-driven strategies to learn depth estimation priors from large scale benchmarks using deep learning models. However, the reconstructed 3D point clouds lack geometry details, due to their discrete representation essence, which makes them not friendly to downstream applications.

More recent methods use implicit functions such as signed distance functions (SDFs) [77] or occupancy functions [51] as continuous representations of 3D shapes and scenes. Using volume rendering, we can learn neural implicit functions by comparing their 2D renderings with multi-view ground truth including color [88, 97], depth [88, 3, 97] or normal [88, 76, 16] maps. Although the supervision of using depth images as rendering target can provide detailed structure information and guide importance sampling along rays [88], both the missing depth at holes and the unawareness of occluded structures make it hard to significantly improve the reconstruction accuracy. Hence, how to more

effectively leverage depth supervision for geometry inference through volume rendering is still a challenge.

To overcome this challenge, we introduce to learn neural implicit through volume rendering with an attentive depth fusion prior. Our key idea is to provide neural networks the flexibility of choosing geometric clues, i.e., the geometry that has been learned and the Truncated Signed Distance Function (TSDF) fused from all available depth images, and combining them into neural implicit for volume rendering. We regard the TSDF as a prior sense of the scene, and enable neural networks to directly use it as a more accurate representation. The TSDF enables accessing the missing depth at holes on one depth image and the occluded structures that are invisible from the current view, which remedies the demerits of using depth ground truth to supervise rendering. To achieve this, we introduce an attention mechanism to allow neural networks to balance the contributions of currently learned geometry and the TSDF in the neural implicit, which leads the TSDF into an attentive depth fusion prior. Our method works with either known camera poses or camera tracking in the context of SLAM, where our prior could be either a one-time fused TSDF that represents a whole scene or an incrementally fused TSDF that represents a partial scene. We evaluate our performance on benchmarks containing synthetic and real-world scans, and report our superiority over the latest methods with known or estimated camera poses. Our contributions are listed below.

We present a novel volume rendering framework to learn neural implicit representations from RGBD images. We enable neural networks to use either currently learned geometry or the one from depth fusion in volume rendering, which leads to a novel attentive depth fusion prior for learning neural implicit functions inheriting the merits of both the TSDF and the inference.

ii) We introduce a novel attention mechanism and a neural network architecture to learn attention weights for the attentive depth fusion prior in neural rendering with either known camera poses or camera pose tracking in SLAM.

iii) We report the state-of-the-art performance in surface reconstruction and camera tracking on benchmarks containing synthetic and real-world scans.

2 Related Work

Learning 3D implicit functions using neural networks has made huge progress [61, 62, 46, 51, 88, 78, 73, 76, 16, 56, 34, 25, 11, 93, 42, 44, 9, 43, 4, 33, 5, 19, 26, 47, 20, 70, 12, 48, 55, 17, 27, 79, 35, 32, 21, 7, 58, 86, 52]. We can learn neural implicit functions from 3D ground truth [23, 8, 54, 45, 68, 39, 69], 3D point clouds [94, 41, 15, 1, 92, 2, 10] or multi-view images [46, 14, 51, 77, 88, 78, 73, 76, 16]. We briefly review methods with multi-view supervision below.

2.1 Multi-view Stereo

Classic multi-view stereo (MVS) [61, 62] employs multi-view photo consistency to estimate depth maps from multiple RGB images. They rely on matching key points on different views, and are limited by large viewpoint variations and complex illumination. Without color, space carving [30] is also an effective way of reconstructing 3D structure as voxel grids.

Recent methods employ data driven strategy to train neural networks to predict depth maps from either depth supervision [84] or multi-view photo consistency [95].

2.2 Neural Implicit from Multi-view Supervision

Early works leverage various differentiable renderers [63, 38, 26, 89, 37, 80, 50, 36, 67] to render the learned implicit functions into images, so that we can measure the error between rendered images and ground truth images. These methods usually require masks to highlight the object, and use surface rendering to infer the geometry, which limits their applications in scenes. Similarly, DVR [50] and IDR [87] predict the radiance near surfaces.

With volume rendering, NeRF [46] and its variations [53, 47, 57, 60, 97, 3, 75, 6, 98, 67] model geometry and color together. They can generate plausible images from novel viewpoints, and do not need masks during rendering procedure. UNISURF [51] and NeuS [77] learn occupancy functions and SDFs by rendering them with colors using revised rendering equations. Following methods

RGB Image

I

Available Depth Images {

D

}

+1

-1

TSDF

Bandwidth

Within

Bandwidth

Outside

Bandwidth

Learnable

Feature Grids

Trilinear

Feature Interpolation

TSDF Interpolation

Attention

Weighting

Depth   Fusion

Known or Estimated

M

...

+

{

Volume Rendering

Rendered RGB

I

Rendered Depth

D

j

j

j

q

V k

q

1

q

2

f

l

f

h

f

c

o

l

o

h

f(

q

)

2

c(

q

)

2

'

j

j

'

Gl

Gh

Gc

t

l

t

h

t

c

f

l

f(

q

)

1

Gl

t

l

f

c

c(

q

)

1

Gc

t

c

Gs

f

a

o

s

improve accuracy of implicit functions using additional priors or losses related to depth [88, 3, 97], normals [88, 76, 16], and multi-view consistency [14].

Depth images are also helpful to improve the inference accuracy. Depth information can guide sampling along rays [88] or provide rendering supervision [88, 3, 97, 96, 24, 31, 82, 98], which helps neural networks to estimate surfaces.

2.3 Neural Implicit with SLAM

Given RGBD images, more recent methods [91, 81, 72, 59] employ neural implicit representations in SLAM. iMAP [66] shows that an MLP can serve as the only scene representation in a realtime SLAM system. NICE-SLAM [97] introduces a hierarchical scene representation to reconstruct large scenes with more details. NICER-SLAM [96] uses easy-to-obtain monocular geometric cues without requiring depth supervision. Co-SLAM [74] jointly uses coordinate and sparse parametric encodings to learn neural implicit functions. Segmentation priors [29, 18] show their potentials to improve the performance of SLAM systems. With segmentation priors, vMAP [29] represents each object in the scene as a neural implicit in a SLAM system.

Instead of using depth ground truth to only supervise rendering, which contains incomplete depth and unawareness of occlusion, we allow our neural network to directly use depth fusion priors as a part of neural implicit, and determine where and how to use depth priors along with learned geometry.

3 Method

Overview. We present an overview of our method in Fig. 1. Given RGBD images { I j , D j } J j =1 from J view angles, where I j and D j denote the RGB and depth images, available either all together or in a streaming way, we aim to infer the geometry of the scene as an occupancy function f which predicts the occupancy f ( q ) at arbitrary locations q = ( x, y, z ) . Our method works with camera poses { M j } that are either known or estimated by our camera tracking method in the context of SLAM.

We learn the occupancy function through volume rendering using the RGBD images { I j , D j } J j =1 as supervision. We render f with a color function c which predicts an RGB color c ( q ) at locations q into an RGB image I ′ j and a depth image D ′ j , which are optimized to minimize their rendering errors to the supervision { I j , D j } .

We start from shooting rays { V k } from current view I j , and sample queries q along each ray V k . For each query q , we employ learnable feature grids G l , G h , and G c covering the scene to interpolate its hierarchical geometry features t l , t h , and a color feature t c by trilinear interpolation. Each feature is further transformed into an occupancy probability or RGB color by their corresponding decoders f l , f h , and f c . For queries q inside the bandwidth of a TSDF grid G s fused from available depth images { D i } , we leverage its interpolation from G s as a prior of coarse occupancy estimation. The prior is attentive by a neural function f a which determines the occupancy function f ( q ) by combining currently learned geometry and the coarse estimation using the learned attention weights from f a .

Lastly, we use the occupancy function f ( q ) and the color function c ( q ) to render I ′ and D ′ through volume rendering. With a learned f , we run the marching cubes [40] to reconstruct a surface.

Depth Fusion and Bandwidth Awareness. With known or estimated camera poses { M j } , we get a TSDF by fusing depth images { D j } that are available either all together or in a streaming way. Our method can work with a TSDF that describes either a whole scene or just a part of the scene, and we will report the performance with different settings in ablation studies. We use the TSDF to provide a coarse occupancy estimation which removes the limit of the missing depth or the unawareness of occlusion on single depth supervision. The TSDF is a grid which predicts signed distances at any locations inside it through trilinear interpolation. The predicted signed distances are truncated with a threshold µ into a range of [ -1 , 1] which covers a band area on both sides of the surface in Fig. 1.

Since the TSDF predicts signed distances for queries within the bandwidth with higher confidence than the ones outside the bandwidth, we only use the depth fusion prior within the bandwidth. This leads to different ways of learning geometries, as shown in Fig. 1. Specifically, for queries q within the bandwidth, we model the low-frequency surfaces using a low resolution feature grid G l , learn the high-frequency details as a complementation using a high resolution feature grid G h , and let our attention mechanism to determine how to use the depth fusion prior from the TSDF G s . Instead, we only use the low resolution feature grid G l to model densities outside the bandwidth during volume rendering. Regarding color modeling, we use the feature grid G c with the same size for interpolating color of queries over the scene.

Feature Interpolation. For queries q sampled along rays, we use the trilinear interpolation to obtain its features h l , h h , and h c from learnable G l , G h , G c and occupancy estimation o s from G s . Both feature grids and the TSDF grid cover the whole scene and use different resolutions, where learnable feature vectors are associated with vertices on each feature grid. Signed distances at vertices on G s may change if we incrementally fuse depth images in the context of SLAM.

Occupancy Prediction Priors. Similar to NICE-SLAM [97], we use pre-trained decoders f l and f h to predict low frequency occupancies o l and high frequency ones o h from the interpolated features t l and t h , respectively. We use f l and f h as an MLP decoder in ConvONet [54] respectively, and minimize the binary cross-entropy loss to fit the ground truth. After pre-training, we fix the parameters in f l and f h , and use them to predict occupancies below,

where we enhance f h by concatenating f l as f h ← [ f h , f l ] and denote o lh as the occupancy predicted at query q by the learned geometry.

Color Predictions. Similarly, we use the interpolated feature t c and an MLP decoder f c to predict color at query q , i.e., c ( q ) = f c ( q , t c ) . We predict color to render RGB images for geometry inference or camera tracking in SLAM. The decoder is parameterized by parameters θ which are optimized with other learnable parameters in the feature grids.

Attentive Depth Fusion Prior. We introduce an attention mechanism to leverage the depth fusion prior. We use a deep neural network to learn attention weights to allow networks to determine how to use the prior in volume rendering.

As shown in Fig. 1, we interpolate a signed distance s ∈ [ -1 , 1] at query q from the TSDF G s using trilinear interpolation, where G s is fused from available depth images. We formulate this interpolation as s = f s ( q ) ∈ [ -1 , 1] . We normalize s into an occupancy o s ∈ [0 , 1] , and regard o s as the occupancy predicted at query q by the depth fusion prior.

For query q within the bandwidth, our attention mechanism trains an MLP f a to learn attention weights α and β to aggregate the occupancy o lh predicted by the learned geometry and the occupancy o s predicted by the depth fusion prior, which leads the TSDF to an attentive depth fusion prior. Hence, our attention mechanism can be formulated as,

We implement f a using an MLP with 6 layers. The reason why we do not use a complex network like Transformer is that we want to justify the effectiveness of our idea without taking too much credit from complex neural networks. Regarding the design, we do not use coordinates or positional encoding as a part of input to avoid noisy artifacts. Moreover, we leverage a Softmax normalization layer to achieve α + β = 1 , and we do not predict only one parameter α and use 1 -α as the second weight, which also degenerates the performance. We will justify these alternatives in experiments.

For query q outside the bandwidth, we predict the occupancy using the feature t l interpolated from the low frequency grid G l and the decoder f l to describe the relatively simpler geometry. In summary, we eventually formulate our occupancy function f as a piecewise function below,

Volume Rendering. We render the color function c and occupancy function f into RGB I ′ and depth D ′ images to compare with the RGBD supervision { I, D } .

With camera poses M j , we shoot a ray V k from view I j . V k starts from the camera origin m and points a direction of r . We sample N points along the ray V k using stratified sampling and uniformly sampling near the depth, where each point is sampled at q n = m + d n r and d n corresponds to the depth value of q n on the ray. Following UNISURF [51], we transform occupancies f ( q n ) into weights w n which is used for color and depth accumulation along the ray V k in volume rendering,

Loss. With known camera pose M j , we render the scene into the color and depth images at randomly sampled K pixels on the j -th view, and optimize parameters by minimizing the rendering errors,

In the Context of SLAM. We can jointly do camera tracking and learning neural implicit from streaming RGBD images. To achieve this, we regard the camera extrinsic matrix M j as learnable parameters, and optimize them by minimizing our rendering errors. Here, we follow [97] to weight the depth rendering loss to decrease the importance at pixels with large depth variance along the ray,

where V ar ( D ′ j ( k )) = ∑ N n =1 w n ( D ′ j ( k ) -d n ) 2 . Moreover, with streaming RGBD images, we incrementally fuse the most current depth image into TSDF G s . Specifically, the incremental fusion includes a pre-fusion and an after-fusion stage. The pre-fusion aims to use a camera pose coarsely estimated by a traditional method to fuse a depth image onto a current TSDF to calculate a rendering error for a more accurate pose estimation at current frame. The after-fusion stage will refuse the depth image onto the current TSDF for camera tracking at the next frame. Please refer to our supplementary materials for more details.

We do tracking and mapping iteratively. For mapping procedure, we render E frames each time and back propagate rendering errors to update parameters. E frames include the current frame and E -1 key frames that have overlaps with the current frame. For simplicity, we merely maintain a key frame list by adding incoming frames with an interval of 50 frames for fair comparisons.

Details. The optimization is performed at three stages, which makes optimization converge better. We first minimize L D by optimizing low frequency feature grid G l , and then both low and high frequency feature grid G l and G h , and finally minimize Eq 5 by optimizing G l , G h , and G c . Our bandwidth from the TSDF G s covers 5 voxels on both sides of the surface. We shoot K = 1000 or 5000 rays for reconstruction or tracking from each view, and render E = 5 or 10 frames each time for fair comparison with other methods. We set λ = 0 . 2 , λ 1 = 0 . 5 in loss functions. We sample N = 48 points along each ray for rendering. More implementation details can be found in our supplementary materials.

4 Experiments and Analysis

4.1 Experimental Setup

Datasets. We report evaluations on both synthetic datasets and real scans including Replica [64] and ScanNet [13]. For fair comparisons, we report results on the same scenes from Replica and ScanNet

as the latest methods. Specifically, we report comparisons on all 8 scenes in Replica. As for ScanNet, we report comparison on scene 50, 84, 580, and 616 used by MonoSDF [88], and also scene 59, 106, 169, and 207 used by NICE-SLAM [97]. We mainly report average results over the dataset, please refer to our supplementary materials for results on each scene.

Metrics. With the learned occupancy function f , we reconstruct the surface of a scene by extracting the zero level set of f using the marching cubes algorithm [40]. Following previous studies [97, 22], we use depth L1 [cm], accuracy [cm], completion [cm], and completion ratio [<5cm % ] as metrics to evaluate reconstruction accuracy on Replica. Additionally, we report Chamfer distance (L1), precision, recall, and F-score with a threshold of 0 . 05 to evaluate reconstruction accuracy on ScanNet. To evaluate the accuracy in camera tracking, we use ATE RMSE [65] as a metric. We follow the same parameter setting in these metrics as [97].

4.2 Evaluations

Evaluations on Replica. We use the same set of RGBD images as [66, 96]. We report evaluations in surface reconstruction and camera tracking in Tab. 1, Tab. 2 and Tab. 3, respectively.

We jointly optimize camera poses and learn geometry in the context of SLAM. The depth fusion prior G s incrementally fuses depth images using estimated camera poses. We report the accuracy of reconstruction from G s as 'TSDF'. Compared to this baseline, we can see that our method improves the reconstruction using the geometry learned through volume rendering and occupancy prediction priors. We visualize the advantage of learning geometry in Fig. 2. Note that the holes in TSDF are caused by the absence of RGBD scans. We can see that our neural implicit keeps the correct structure in TSDF and plausibly completes the missing structures in TSDF. We further visualize the attention weights β learned for depth fusion in Fig. 3. We visualize the cross sections on 4 scenes, where β learned in bandwidth are shown in color red. Generally, the neural implicit mostly focuses

TSDF

Ours

more on the depth fusion priors in areas where TSDF is complete, while focusing more on the learned geometry in areas where TSDF is incomplete. In the area where TSDF is complete, the network also pays some attention to the inferred occupancy because the occupancy interpolated from TSDF may not be accurate, especially on the most front of surfaces in Fig. 3.

Moreover, our method outperforms the latest implicit-based SLAM methods like NICE-SLAM [97] and NICER-SLAM [96]. We present visual comparisons in Fig. 6, where our method produces more accurate and compact geometry. For method using GT camera poses like vMAP [29] and MonoSDF [88], we achieve much better performance as shown by 'Ours*' in Tab. 1 and 'Ours' in Tab. 2. Although we require the absolute dense depth maps, compared to MonoSDF [88] that can work with

(a)

(b)

(c)

(d)

͗Ͳ΁͐ͼ͹ͼͿͺͮͽ͓΂ͻͰ΁Ͷͼͻ̻ͽͻʹ

1

0

scaled depth maps, our method only can use the views in front of the current time step, where MonoSDF [88] can utilize all views during the whole training process.

In camera tracking, our results in Tab. 3 achieve the best in average. We compare the estimated trajectories on scene off-4 in Fig. 4 (c). The comparison shows that our attentive depth fusion prior can also improve the camera tracking performance through volume rendering.

Evaluations on ScanNet. We further evaluate our method on ScanNet. For surface reconstruction, we compare with the latest methods for learning

NICE-SLAM

Ours

Scene 59 (ScanNet)

Scene 169  (ScanNet)

(a)

(b)

(c)

neural implicit from multiview images. We report both their results and ours with GT camera poses, where we also fuse every 10 depth images into the TSDF G s and render every 10 frames for fair comparisons. Numerical comparisons in Tab. 4 show that our method achieves the best performance in terms of all metrics, where we use the culling strategy introduced in MonoSDF [88] to clean the reconstructed mesh. We highlight our significant improvements in visual comparisons in Fig. 7. We see that our method can reconstruct sharper, more compact and detailed surfaces than other methods. We detail our comparisons on every scene with the top results reported by GO-Surf [75] and NICE-SLAM [97] in Tab. 5 and Tab. 6. The comparisons in Tab. 5 show that our method achieves higher reconstruction accuracy while GO-Surf produces more complete surfaces. We present visual comparisons with error maps in Fig. 5.

In camera tracking, we compare with the latest methods. We incrementally fuse the most current depth frames into TSDF G s which is used for attentive depth fusion priors in volume rendering. Numerical comparisons in Tab. 7 show that our results are better on 2 out of 4 scenes and achieve the best in average. Tracking trajectory comparisons in Fig. 4 (a) and (b) also show our superiority.

NICE-SLAM

GO-Surf

Ours

Room-0

NICE-SLAM

Ours

Off-3

NICE-SLAM

Ours

Neus

ManhattanSDF

Neuris

Ours

GT

4.3 Ablation Studies

We report ablation studies to justify the effectiveness of modules in our method on Replica. We use estimated camera poses in ablation studies.

Effect of Depth Fusion. We first explore the effect of different ways of depth fusion on the performance. According to whether we use GT camera poses or incremental fusion, we try 4 alternatives with G s obtained by fusing all depth images at the very beginning using GT camera poses (Offline) or fusing the current depth image incrementally (Online), and using tracking to estimate camera poses (Tracking) or GT camera poses (GT) in Tab. 8. The comparisons show that our method can work well with either GT or estimated camera poses and fusing depth either all together or incrementally in a streaming way. Additional conclusion includes that GT camera poses in either depth fusion and rendering do improve the reconstruction accuracy, and the structure fused from more recent frames is more important than the whole structure fused from all depth images.

Effect of Attention. We explore the effect of attention on the performance in Tab. 9. First of all, we remove the attention mechanism, and observe a severe degeneration in accuracy, which indicates that the attention plays a big role in directly applying depth fusion priors in

neural implicit. Then, we try to apply attention of depth fusion priors on geometries predicted by different features including low frequency t l , high frequency t h , and both of them. The comparisons show that adding details from depth fusion priors onto low frequency surfaces does not improve the performance. More analysis on attention mechanism can be found in Fig. 9.

Attention Alternatives. Our preliminary results show that using softmax normalization layer achieves better performance than other alternatives. Since we merely need two attention weights α and β to combine the learned geometry and the depth fusion prior, one alternative is to use a sigmoid function to predict

one attention weight and use its difference to 1 as another attention weight. However, sigmoid can not effectively take advantage of the depth fusion prior. We also try to use coordinates as an input, which pushes the network to learn spatial sensitive attentions. While the reconstructed surfaces turn out to be noisy in Fig. 8. This indicates that our attention learned a general attentive pattern for all locations in the scene. We also report the numerical comparisons in Tab. 10. Please refer to our supplementary materials for more results related to attention alternatives.

NoAttention

Sigmoid

Coordinates

Ours

Effect of Bandwidth. We further study the effect of bandwidth on the performance in Tab. 11. We try to use attentive depth fusion priors everywhere in the scene with no bandwidth. The degenerated results indicate that the truncated area outside the bandwidth does not provide useful structures to improve the performance. Moreover, wider bandwidth may cause more artifacts caused by the calculation of the TSDF G s while narrower bandwidth brings more incomplete structures, neither of which improves the performance. Instead of using low frequency geometry outside the bandwidth, we also try to use high frequency surfaces. We can see that low frequency geometry is more suitable to describe the scene outside the bandwidth.

Effect of Prediction Priors. Prediction priors from decoders f l and f h are also important for accuracy improvements. Instead of using fixed parameters in these decoders, we try to train f l (Fix f h ) or f h (Fix f l ) with other parameters. Numerical comparisons in Tab. 12 show that

training f l or f h cannot improve the performance, or even fails in camera tracking (Fix f h ).

More Analysis on How Attention Works. Additionally, we do a visual analysis on how attention works in Fig. 9. We sample points on the GT mesh, and get the attention weights in Fig. 9 (a). At each point, we show its distance to the mesh from the TSDF in Fig. 9 (b) and the mesh from the inferred occupancy in Fig. 9 (c), respectively. Fig. 9 (d) indicates where the former is smaller than the latter. The high correlation between Fig. 9 (a) and Fig. 9 (d) indicates that the attention network focuses more on the occupancy producing smaller errors to the GT surface. Instead, the red in Fig. 9 (e) indicates where the interpolated occupancy is larger than the inferred occupancy is not correlated to the one in Fig. 9 (a). The uncorrelation indicates that the attention network does not always focus on the larger occupancy input but the one with smaller errors, even without reconstructing surfaces.

(a) Attention on TSDF

(b) Error distances to

the mesh from TSDF

(c) Error distances to the mesh

from inferred occupancies

(d) Binary map for

smaller errors (red) of TSDF

(e) Binary map for

larger (red) interpolated occupancies

5 Conclusion

We propose to learn neural implicit through volume rendering with attentive depth fusion priors. Our novel prior alleviates the incomplete depth at holes and the unawareness of occluded structures when using depth images as supervision in volume rendering. We also effectively enable neural networks to determine how much depth fusion prior can be directly used in the neural implicit. Our method can work well with depth fusion from either all depth images together or the ones available in a streaming way, using either known or estimated camera poses. To this end, our novel attention successfully learns how to combine the learned geometry and the depth fusion prior into the neural implicit for more accurate geometry representations. The ablation studies justify the effectiveness of our modules and training strategy. Our experiments on benchmarks with synthetic and real scans show that our method learns more accurate geometry and camera poses than the latest neural implicit methods.

References

Matan Atzmon and Yaron Lipman. SAL: Sign agnostic learning of shapes from raw data. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Matan Atzmon and yaron Lipman. SALD: sign agnostic learning with derivatives. In International Conference on Learning Representations , 2021.

Dejan Azinovi´ c, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition , pages 6290-6301, 2022.

Ma Baorui, Liu Yu-Shen, and Han Zhizhong. Reconstructing surfaces for sparse point clouds with onsurface priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Ma Baorui, Han Zhizhong, Liu Yu-Shen, and Zwicker Matthias. Neural-pull: Learning signed distance functions from point clouds by learning to pull space onto surfaces. In International Conference on Machine Learning (ICML) , 2021.

Aljaz Bozic, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Niessner. Transformerfusion: Monocular rgb scene reconstruction using transformers. Advances in Neural Information Processing Systems , 2021.

Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR , 2023.

Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard A. Newcombe. Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In European Conference on Computer Vision , volume 12374, pages 608-625, 2020.

Chao Chen, Zhizhong Han, and Yu-Shen Liu. Unsupervised inference of signed distance functions from single sparse point clouds without learning priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Chao Chen, Yu-Shen Liu, and Zhizhong Han. Latent partition implicit with surface codes for 3d representation. In European Conference on Computer Vision , 2022.

Chao Chen, Yu-Shen Liu, and Zhizhong Han. Gridpull: Towards scalability in learning implicit representations from 3d point clouds. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2023.

Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In The Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. CoRR , abs/1702.04405, 2017.

Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. Geo-Neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction. In Advances in Neural Information Processing Systems , 2022.

Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 3789-3799, 2020.

Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, and Xiaowei Zhou. Neural 3d scene reconstruction with the manhattan-world assumption. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas O˘ guz. 3dgen: Triplane latent diffusion for textured mesh generation, 2023.

Yasaman Haghighi, Suryansh Kumar, Jean-Philippe Thiran, and Luc Van Gool. Neural implicit dense semantic slam, 2023.

Zhizhong Han, Guanhui Qiao, Yu-Shen Liu, and Matthias Zwicker. Seqxy2seqz: Structure learning for 3d shapes by sequentially predicting 1d occupancy segments from 2d coordinates, 2020.

Ayaan Haque, Matthew Tancik, Alexei A. Efros, Aleksander Holynski, and Angjoo Kanazawa. Instructnerf2nerf: Editing 3d scenes with instructions, 2023.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-toprompt image editing with cross attention control. 2022.

Jiahui Huang, Shi-Sheng Huang, Haoxuan Song, and Shi-Min Hu. Di-fusion: Online implicit 3d reconstruction with deep priors, 2021.

Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3D scenes. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Hanqi Jiang, Cheng Zeng, Runnan Chen, Shuai Liang, Yinhe Han, Yichao Gao, and Conglin Wang. Depth-neus: Neural implicit surfaces learning for multi-view reconstruction based on depth information optimization. 2023.

Sijia Jiang, Jing Hua, and Zhizhong Han. Coordinate quantized neural implicit representations for multi-view 3d reconstruction. In IEEE International Conference on Computer Vision , 2023.

Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. SDFDiff: Differentiable rendering of signed distance fields for 3D shape optimization. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Leonid Keselman and Martial Hebert. Flexible Techniques for Differentiable Rendering with 3D Gaussians, 2023.

Lukas Koestler, Nan Yang, Niclas Zeller, and Daniel Cremers. TANDEM: tracking and dense mapping in real-time using deep multi-view stereo. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Conference on Robot Learning , volume 164, pages 34-45, 2021.

Xin Kong, Shikun Liu, Marwan Taher, and Andrew J Davison. vmap: Vectorised object mapping for neural field slam. arXiv preprint arXiv:2302.01838 , 2023.

A. Laurentini. The visual hull concept for silhouette-based image understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence , 16(2):150-162, 1994.

Seunghwan Lee, Gwanmo Park, Hyewon Son, Jiwon Ryu, and Han Joo Chae. Fastsurf: Fast neural rgb-d surface reconstruction using per-frame intrinsic refinement and tsdf fusion prior learning. arXiv preprint arXiv:2303.04508 , 2023.

Jiabao Lei, Jiapeng Tang, and Kui Jia. Rgbd2: Generative scene synthesis via incremental view inpainting using rgbd diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2023.

Tianyang Li, Xin Wen, Yu-Shen Liu, Hua Su, and Zhizhong Han. Learning deep implicit functions for 3d shapes with dynamic code clouds. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and ChenHsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Zhihao Liang, Zhangjin Huang, Changxing Ding, and Kui Jia. Helixsurf: A robust and efficient neural implicit surface learning of indoor scenes with iterative intertwined regularization, 2023.

Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey. SDF-SRN: Learning signed distance 3D object reconstruction from static images. In Advances in Neural Information Processing Systems , 2020.

Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3D supervision. In Advances in Neural Information Processing Systems , 2019.

Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. DIST: Rendering deep implicit signed distance function with differentiable sphere tracing. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Shi-Lin Liu, Hao-Xiang Guo, Hao Pan, Pengshuai Wang, Xin Tong, and Yang Liu. Deep implicit moving least-squares functions for 3D reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3D surface construction algorithm. Computer Graphics , 21(4):163-169, 1987.

Baorui Ma, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Neural-pull: Learning signed distance functions from point clouds by learning to pull space onto surfaces. In International Conference on Machine Learning , 2021.

Baorui Ma, Yu-Shen Liu, and Zhizhong Han. Learning signed distance functions from noisy 3d point clouds via noise to noise mapping. In International Conference on Machine Learning (ICML) , 2023.

Baorui Ma, Yu-Shen Liu, Matthias Zwicker, and Zhizhong Han. Surface reconstruction from point clouds by learning predictive context priors. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Baorui Ma, Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Towards better gradient consistency for neural signed distance functions via level set alignment. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. ACORN: adaptive coordinate networks for neural scene representation. CoRR , abs/2105.02788, 2021.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision , 2020.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. arXiv:2201.05989 , 2022.

Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8280-8290, June 2022.

Richard A. Newcombe, Steven J. Lovegrove, and Andrew J. Davison. Dtam: Dense tracking and mapping in real-time. In International Conference on Computer Vision , pages 2320-2327, 2011.

Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In International Conference on Computer Vision , 2021.

Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam. isdf: Real-time neural signed distance fields for robot perception. In Robotics: Science and Systems , 2022.

Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. IEEE International Conference on Computer Vision , 2021.

Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision , 2020.

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv , 2022.

Radu Alexandru Rosu and Sven Behnke. Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Darius Rückert, Linus Franke, and Marc Stamminger. Adop: Approximate differentiable one-pixel point rendering. arXiv:2110.06635 , 2021.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023.

Erik Sandström, Kevin Ta, Luc Van Gool, and Martin R. Oswald. Uncle-slam: Uncertainty learning for dense neural slam, 2023.

Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.

Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision , 2016.

Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3D-structure-aware neural scene representations. In Advances in Neural Information Processing Systems , 2019.

Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard A. Newcombe. The replica dataset: A digital replica of indoor spaces. CoRR , abs/1906.05797, 2019.

Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 573-580, 2012.

Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison. imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6229-6238, 2021.

Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. NeuralRecon: Real-time coherent 3D reconstruction from monocular video. IEEE Conference on Computer Vision and Pattern Recognition , 2021.

Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

Jiapeng Tang, Jiabao Lei, Dan Xu, Feiying Ma, Kui Jia, and Lei Zhang. SA-ConvONet: Sign-agnostic optimization of convolutional occupancy networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng. Delicate textured mesh recovery from nerf via adaptive surface refinement. arXiv preprint arXiv:2303.02091 , 2022.

Zachary Teed and Jia Deng. DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. Advances in neural information processing systems , 2021.

Andreas L. Teigen, Yeonsoo Park, Annette Stahl, and Rudolf Mester. Rgb-d mapping and tracking in a plenoxel radiance field, 2023.

Delio Vicini, Sébastien Speierer, and Wenzel Jakob. Differentiable signed distance function rendering. ACM Transactions on Graphics , 41(4):125:1-125:18, 2022.

Hengyi Wang, Jingwen Wang, and Lourdes Agapito. Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam, 2023.

Jingwen Wang, Tymoteusz Bleja, and Lourdes Agapito. Go-surf: Neural feature grid optimization for fast, high-fidelity rgb-d surface reconstruction. In International Conference on 3D Vision , 2022.

Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, and Wenping Wang. NeuRIS: Neural reconstruction of indoor scenes using normal priors. In European Conference on Computer Vision , 2022.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advances in Neural Information Processing Systems , pages 27171-27183, 2021.

Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. HF-NeuS: Improved surface reconstruction using high-frequency details. 2022.

Chao-Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari. Multiview compressive coding for 3d reconstruction, 2023.

Yunjie Wu and Zhengxing Sun. DFR: differentiable function rendering for learning 3D generation from images. Computer Graphics Forum , 39(5):241-252, 2020.

Liu Xinyang, Li Yijin, Teng Yanbin, Bao Hujun, Zhang Guofeng, Zhang Yinda, and Cui Zhaopeng. Multi-modal neural radiance field for monocular dense slam with a light-weight tof sensor. In International Conference on Computer Vision (ICCV) , 2023.

Xiangyu Xu, Lichang Chen, Changjiang Cai, Huangying Zhan, Qingan Yan, Pan Ji, Junsong Yuan, Heng Huang, and Yi Xu. Dynamic voxel grid optimization for high-fidelity rgb-d supervised surface reconstruction, 2023.

Xingrui Yang, Hai Li, Hongjia Zhai, Yuhang Ming, Yuqian Liu, and Guofeng Zhang. Vox-fusion: Dense tracking and mapping with voxel-based neural implicit representation. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) , Dec 2022.

Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. European Conference on Computer Vision , 2018.

Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Advances in Neural Information Processing Systems , 2021.

Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall. Bakedsdf: Meshing neural sdfs for real-time view synthesis. arXiv , 2023.

Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems , 33, 2020.

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction. ArXiv , abs/2022.00665, 2022.

Sergey Zakharov, Wadim Kehl, Arjun Bhargava, and Adrien Gaidon. Autolabeling 3D objects with differentiable rendering of sdf shape priors. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In CVPR , 2017.

Youmin Zhang, Fabio Tosi, Stefano Mattoccia, and Matteo Poggi. Go-slam: Global optimization for consistent 3d instant reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023.

Wenbin Zhao, Jiabao Lei, Yuxin Wen, Jianguo Zhang, and Kui Jia. Sign-agnostic implicit learning of surface self-similarities for shape modeling and reconstruction from raw point clouds. CoRR , abs/2012.07498, 2020.

Junsheng Zhou, Baorui Ma, Shujuan Li, Yu-Shen Liu, and Zhizhong Han. Learning a more continuous zero level set in unsigned distance fields through level set projection. In Proceedings of the IEEE/CVF international conference on computer vision , 2023.

Junsheng Zhou, Baorui Ma, Yu-Shen Liu, Yi Fang, and Zhizhong Han. Learning consistency-aware unsigned distance functions progressively from raw point clouds. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In IEEE Conference on Computer Vision and Pattern Recognition , pages 6612-6619, 2017.

Zihan Zhu, Songyou Peng, Viktor Larsson, Zhaopeng Cui, Martin R. Oswald, Andreas Geiger, and Marc Pollefeys. NICER-SLAM: neural implicit scene encoding for RGB SLAM. CoRR , abs/2302.03594, 2023.

Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Zi-Xin Zou, Shi-Sheng Huang, Yan-Pei Cao, Tai-Jiang Mu, Ying Shan, and Hongbo Fu. Mononeuralfusion: Online monocular neural 3d reconstruction with geometric priors. CoRR , abs/2209.15153, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of 3D scene reconstruction from posed RGB-D images by learning an implicit occupancy function.\n\nUnlike previous methods [58, 3, 65, Ref_DS] that directly use the depth values (obtained either from depth images, SfM pipelines, or from depth prediction networks) of individual rays (i.e. pixels) as supervisory signal leading to learning better scene geometry, faster convergence [Ref_DS, Ref_IN] and inference [Ref_IN, 58], this method proposes to first fuse the depth frames into a TSDF representation using some off-the-shelf method and then query the TSDF for obtaining the occupancy value corresponding to a particular point on a sampled ray for supervision.In other words, the off-the-shelf obtained TSDF of the scene is used as a scene prior.\n\nThe motivation for relying on TSDF representation of the scene is that the depth images obtained from RGBD sensors suffer from missing depth values (or holes) which makes it challenging to supervise 3D point samples corresponding to those rays.\n\nThe primary contribution of this paper is to learn an attention function that, for a 3d point sample,  estimates the weights for the linear combination of occupancy values which are predicted from the neural network and occupancy values obtained from the TSDF scene prior (as in, final_occupancy = w1*predicted_occupancy + w2*TSDF_obtained_occupancy_prior).  In this sense, the TSDF prior becomes an attentive depth fusion prior.\n\nThe network is supervised using volumetric rendering of depth and colour as proposed in [31].\n\nJust like previous methods (e.g. [48, 58]), this method too relies on a hybrid representation: three discrete feature grids are learned, two for low and high-resolution representation of geometry and one for colour, and the tri-linearly interpolated feature value for a corresponding 3D point is decoded by corresponding MLP decoders to obtain occupancy and colour for that 3D point.\n\nThe truncated nature of TSDF (it belongs to (-1 , 1) ) is also exploited to inform when to use low-resolution grid features to get occupancy and when to rely on both low and high-resolution grid features to predict occupancy and fuse it (using learned weights) with the TSDF informed occupancy prior; it is clear that points which have TSDF values under the range of 1 or -1 are near surfaces and hence higher resolution feature grids are necessary and in addition, TSDF will explain such points the best.\n\nThe method is also applicable for SLAM settings where RGB-D frames are obtained incrementally in which the camera extrinsics, i.e. poses, are also optimized with the network parameters.\n\n\n[Ref_IN] Kim, Mijeong, Seonguk Seo, and Bohyung Han. \"Infonerf: Ray entropy minimization for few-shot neural volume rendering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[Ref_DS] Deng, Kangle, et al. \"Depth-supervised nerf: Fewer views and faster training for free.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\nSTRENGTHS:\n1. The paper, barring a few places, is well written.\n\n2. This paper has identified the limitation in supervising neural networks that learn implicit functions for dense 3D scene reconstruction using depth images which is difficult in supervising predictions for points whose depth values are not available as they belong to holes in the depth images.\n\n3. Authors have put in a reasonable effort to analyse the proposed method with good qualitative and quantitative analysis.\n\nWEAKNESSES:\n1. The paper lacks any substantial contribution\n\nExplanation:\n\na. It relies on hybrid representation for learning implicit functions for dense 3D reconstruction which has been very well explored before in the same context ([48], [58], [19])\n\nb. The use of pre-trained MLP decoders for predicting occupancy from trilinearly interpolated feature vectors from multi-resolution grids has been inspired by NICE-SLAM[65].\n\nc. The TSDF for the scene is obtained by fusing the posed depth images using the off-the-shelf method.\n\nd. The only contribution of this paper is towards learning an attention function that predicts the weights which are used to linearly fuse the predicted occupancies and occupancy obtained from a TSDF prior.\n\n\n2. Given the way the attention function is learned which is the primary contribution of this work, it almost looks unnecessary :\n\nExplanation: As explained in the method section and also schematically shown in the pipeline figure (fig 1), the attention function takes the sum of the predicted low and high-resolution grid occupancies and occupancy obtained by TSDF prior (normalized between 0 and 1) to predict the weights for the linear combination of the two (eq. 2, eq 3., figure 1.).\n\nThis means the attention function only chooses to learn the weights (alpha and beta, eq. 2, 3) by looking at only two floating point numbers which are between 0 and 1 based on errors in RGB and depth renderings. The RGB and Depth renderings are obtained from the final linear combination of the two occupancies (eq. 2, 3) and their quality would be inversely proportional to the rendering errors.  I don't see what else the attention function can learn to pay attention to other than which one of the inputs is larger.\n\nIn which case, how would it be any different from simply using the softmax operator to combine the two occupancies instead of learning two weights? \n\nHad the attention function taken any feature from grids to reason about how well the network can represent the scene , it could make better decision as which is more important, predicted occupancy or TSDF prior occupancy. Accordingly, it would predict weights for the fusion of the two. \n\n3. The method is affected by the very problem it is trying to solve:\nExplanation: The paper claims that missing depth values are a problem when using them raw for learning implicit functions for scene reconstruction. The fused TSDF prior would help mitigate that. However, from Figure 2 it is clear that TSDF prior could also have holes where the network has to rely on its own learned occupancy. This shows that the idea of using TSDF is itself affected by the very problem it is trying to mitigate.\n\n4. The paper also proposes that the missing depth due to occlusion leads to difficulty while supervising using depth images. However, when posed RGB-D frames of the scene are available, the points that are occluded in one frame would be visible in some other frames anyway. Consider, for example, the classic NeRF [27] paper or [Ref_DS], the novel view renderings are not affected by occlusions in different frames as the occluded ones are anyway visible in other views.\n\n5. The proposed method is very similar to [19] in the sense that both rely on fused TSDF of the scene then it would be interesting to see how the proposed method fairs with [19]. The qualitative results of [19] look quite similar to the ones shown in this paper which motivates me to also consider a quantitative comparison between the two.\n\n\n6. The paper points the reviewer to refer to supplemental material for more results. But there is no supplemental material pdf for reference (the supplemental material also has the paper in it).\n\n\n[Ref_IN] Kim, Mijeong, Seonguk Seo, and Bohyung Han. \"Infonerf: Ray entropy minimization for few-shot neural volume rendering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[Ref_DS] Deng, Kangle, et al. \"Depth-supervised nerf: Fewer views and faster training for free.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\nQUESTIONS:\n1.  How is the proposed attentive depth fusion prior different from simply using a softmax operation over the predicted occupancies and the occupancy obtained from TSDF prior? This question is motivated from the fact that the attention function only takes the two occupancies as input .\n\n2. How is the error in TSDF which is accumulated due to progressive fusion of depth images using their recent estimate of pose handled?\n\nDetail question: In a SLAM setting where we do not have all the RGB-D images beforehand, the TSDF is performed by fusing the scene incrementally using the estimated poses. However, pose estimates from any front-end would have some degree of error. This error would propagate to the TSDF scene prior which is used to construct the final occupancy. The network however is optimized along with camera poses but the TSDF does not seem to be updated based on the optimized camera poses. If, however, it is updated, would it not mean that at every mapping phase, we will have to fuse the entire TSDF again? This would again mean that we will have to keep appropriate number of depth frames for this. However this method, like vMAP[17], iMAP[41], stores keyframes, whose main purpose is for replay while learning incrementally to tackle the catastrophic forgetting issue. No reference to this is provided in the paper.\n\n3. How would the result look like if instead of a linear combination of the predicted occupancy and occupancy from TSDF prior, we use the one with the maximum weight? Could the results indicate that the attention function is redundant? This would also demonstrate if there is any merit in the idea of using TSDF as a prior for composing the occupancies or learning to predict TSDF from TSDF prior is a better idea; [19] has three stages of training, the first is devoted to this.\n\nLIMITATIONS:\nThe authors have not discussed any limitations. However, the following are the two strong limitations of this work:\n\n1. One strong limitation of this work is that it would require the availability of TSDF prior even in test time. So what is the point of learning implicitly? The method is also capable of optimizing the camera poses and hence the resulting occupancy would be better (as also shown in the results). But then how is it different from running a classic bundle adjustment and then fusing the optimized depth values into a more accurate TSDF using optimized camera poses? Whereas methods like MonoSDF[58], FastSURF[19] do not need fused TSDF in the test time and the results are quite comparable if not better.\n\n2. Ideally, the attention function could be also provided with the features from the grids so that the function could learn to decide upon the fusion weights also based on the features which are also being optimized. In this process, the learnt features would be considered when deciding the weights required for fusing the occupancies. This would help the network to actually reason about the fusion based on the geometric understanding learned so far.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a simple yet effective approach of neural implicit surface reconstruction from RGB-D seuqences that leverages the reconstructed TSDF grids as a prior which effectively improve the reconstruction quality of fitting a neaural implicit surface from multi-view RGB-D images directly. The idea of using fused TSDF grids as a geometric prior is novel and makes sense as it comes for free from RGB-D sequences. The proposed attention mechnism of fusing the results from TSDF prior and neural implicit function is also novel and effective through experimental results. The authors have also demonstarted the effectiveness of their proposed methods and each of its components through extensive experiments, ablations and analysis. Except from some very minor issues, this is a good paper. For these reasons, I would reccomend \"Weak Accept\" at this stage.\n\nSTRENGTHS:\n1. The idea of using fused TSDF grids as a prior is novel. Normally, traditional TSDF fusion could achieve better accuracy in geometry reconstruction but falls short of filling the holes in occluded or unobserved regions. Neual implicit methods, on the other hand, is better at hole filling at the cost of slightly less accurate surfaces. Therefore, using TSDF fusion as a prior and combining these two methods is very natual and has very clear motivation. Also as the authors showed, fused TSDF grid already encoded all historical frames and is occlusion-aware, which could also bring benefits to directly fitting to multi-view RGBD images. Moreover, this prior also comes with no extra cost. The proposed learnable attention module to fuse the output from two sources is also a \"simple-yet-effective\" solution.\n\n2. Overall, the quantitative experiments are conducted extensively and are sufficient to show the effectiveness of the proposed methods. Quantitative results show the proposed method achieves SOTA  reconstruction and tracking results compared to previous RGB-D neural implicit SLAM/surface reconstruction methods. Qualitative results also look promising and accompanied with detailed and clear analysis. Ablation studies are also well conducted and justified the effectiveness of each module and design choices.\n\n3. The paper is very well written and easy to follow. The introduction and related work sections are very clear and show clear motivation of the proposed method.\n\n4. The method could run both in an online SLAM manner and under batched offline setting, which increases the system's practical value.\n\nWEAKNESSES:\n1. Overall, the comparisons with RGB-D baselines seems convincing, but some of the experiments in Sec. 4.2 are comparing against baseline methods that rely on RGB-only input. For example in Tab. 2, 4 and Fig. 5. These are not completely fair comparisons as the proposed methods take measured depth directly. Or did you change some settings in the experiments, like supervising the proposed method with predicted depth instead of measured depth?\n\n2. It would be good to also include performance analysis, such as memory and run-time comparison to previous methods, especially for the SLAM setting.\n\nQUESTIONS:\nSome minor questions:\n\n1. What's the resolution and voxel size used in the feature grids and TSDF grids?\n\n2. Regarding the learnable attention module $f_a(\\cdot)$, seems that it is only conditioned on the two occupancy predictions? Have you tried with more conditions?\n\n3. When evaluating reconstruction quality on ScanNet (Tab. 4), what mesh culling strategy did you use? Also is it consistent across all the methods?\n\n4. In the ablation studies on depth fusion (Tab. 8), why would there be difference between GT All and GT? If I understand correctly, the difference is just that GT All operates in a batched manner while GT is online. But this shouldn't cause difference when you run TSDF-fusion?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a pipeline for estimating scene geometry with TSDF represented with neural implicit function, based on multi-view RGBD inputs. The main novelty is a fusion mechanism which utilized fused depth geometry as prior, and fuses geometry prior with estimated geometry using attention-based weighting. The weighting also considers bandwidth of the TSDF representation, allowing for multi-resolution grid features to be leveraged. The results demonstrated superior performance w.r.t. geometry estimation when compared to methods like MonoSDF, and better pose estimation when compared to methods like NICE-SLAM.\n\nSTRENGTHS:\n[1] The idea to fuse input geometry and estimated geometry, instead of using input geometry as constraints when optimizing estimated geometry, is in general novel and sound. The reasoning is, input geometry from fused depth maps may present high confidence in certain areas, and estimated geometry via MLPs may excel at other regions. By fusing these two instead of estimating all regions with MLPs, the method yields better geometry, and benefits pose estimation in SLAM applications. The insights here would be useful to the community to inspire better ways to leverage input geometry when training MLP-based TSDF fields.\n\n[2] Extensive evaluation. The methods evaluates the method w.r.t. geometry estimation with state-of-the-art methods like MonoSDF, which is the main focus of the paper. Beyond geometry, the method also shows improved geometry benefits downstream tasks of pose estimation, and is able show superior performance over SLAM systems with implicit neural representations. Extensive ablation study is also applaudable, which justifies certain design choices,  as well as providing visualization of what is learned with the attention weights.\n\n[3] The paper is in general well-written and easy to follow.\n\nWEAKNESSES:\n[1] Further clarify and justifications are needed in various parts of the paper. \n\n[1.1] For example, on why Transformers are not used to model the attentions, the paper gives an unconvincing reason that people may conjecture that the performance improvements are mainly contributed to Transformers. However, it is unfair and premature to say so without providing actual results with a Transformer-base design. With those additional results, we may then get a better idea on how important the architecture design of the attention mechanism is to the performance gain. Additionally, design choices mentioned in Line 165-169 require justifications via ablation study.\n\n[1.2] Additionally, on the explainability of the learned attention maps, the paper mentions in Line 246 that, 'some area' are paid more attention to, which is not sufficient. Further analysis onto what part of the input geometry is prioritized, and what part is mostly learned, will give better insights of the attention weights, and better justifies the motivation of the proposed. Without further analysis, the attention machoism behaves more or less as a black box.\n\n[1.3] Finally, additional details of the experiment setting are needed. For example, in comparing against MonoSDF, it is not clear whether input depth or estimated depth (with DPT) are used. In experiments where poses are optimized, it is not clear what the initializations look like, what pose representation and regularization are used, and what was the convergence.\n\n[2] Paper writing. For example, the introduction section repeatedly mentions the core contribution of 'attentive depth fusion priors', without giving any explanation or hints of what it exactly is. As a result, the contribution #2 is essentially explaining the 'attentive depth fusion priors', however it is not immediately clear that the 'attentive depth fusion priors' mentioned in contribution #1 is basically explained in #2. Additionally, more discussion should go into explaining import baselines like MonoSDF and NICE-SLAM, on how are they different from the proposed method, so as to better clarify the contributions, and to provide further insights on why the proposed method performs better.\n\nQUESTIONS:\nThere are questions included in my reviews, mostly in Weakness 1.2 and 1.3. it would be great if the authors can address those questions.\n\nLIMITATIONS:\nNot applicable. No potential negative societal impacts are mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve the performance of 3D reconstruction from multi-view RGB-D images. The key innovation is the attentive depth fusion prior, which allows the networks to directly use the depth fusion prior with the inferred occupancy as the learned implicit function. Experiments show that the proposed method work for both one-time fused TSDF and incrementally fused TSDF.\n\nSTRENGTHS:\n1. This paper aims to improve 3D reconstruction by better handling cases such as incomplete depth at holes. This is one limitation of existing work and is an important task.\n\n2. The method section is clearly and well explained to help readers understand the details.\n\n3. The authors have included video results that can help better understand the visual result quality of the proposed method.\n\n4. Extensive comparison and ablation studies have been done to show improvements over prior work and justify the decision choices of all components.\n\nWEAKNESSES:\n1. L242 claims that the proposed method \"plausibly completes the missing structures from TSDF\" in Fig. 2. However, the completed surface by the proposed method seems to be of very low-quality. It seems a simple non-learning mesh hole filling algorithm on top of the meshes on the left of Fig. 2 can produce much cleaner results. Learning-based methods [A,B] that predict TSDF for indoor scene completion can also achieve much better results. \n\n[A] Dai et al. SG-NN: Sparse generative neural networks for self-supervised scene completion of rgb-d scans. CVPR 2020.\n[B] Dai et al. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. CVPR 2018.\n\n2. It is great that the paper has conducted extensive quantitative experiments. However, there are currently 12 tables in the main paper which seems a bit overwhelming. Indeed, some tables are not that critical and can be moved to supplementary. Alternatively, the authors can only present the scene average evaluation in the main paper for some table and move the per-scene results to the supplementary. The new empty space can be instead used to address some limitations.\n\nQUESTIONS:\n1. Since the proposed method is predicting multiple values for the same query points, what is the time increase over a naive neural implicit function (predicting a single signed distance for each query point)?\n\n2. Minor editorial suggestions: \nL173: it would be better to swap \"color function c\" and \"occupancy function f\" for better correspondence.\nL191: includes -> include\nL303: attentions -> attention\n\nLIMITATIONS:\nThe authors have not adequately addressed the limitations or potential negative societal impact. The authors could show some failures cases that indicates some common patterns where the proposed method would fail on", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}