Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors
Authors: Pengchong Hu
Zhizhong Han
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D Reconstruction
SDF
Neural Rendering
Implicit Representations
SLAM

Abstract: Learning neural implicit representations has achieved remarkable performance in 3D reconstruction from multi-view images. Current methods use volume rendering to render implicit representations into either RGB or depth images that are supervised by the multi-view ground truth. However, rendering a view each time suffers from incomplete depth at holes and unawareness of occluded structures from the depth supervision, which severely affects the accuracy of geometry inference via volume rendering. To resolve this issue, we propose to learn neural implicit representations from multi-view RGBD images through volume rendering with an attentive depth fusion prior. Our prior allows neural networks to sense coarse 3D structures from the Truncated Signed Distance Function (TSDF) fused from all available depth images for rendering. The TSDF enables accessing the missing depth at holes on one depth image and the occluded parts that are invisible from the current view. By introducing a novel attention mechanism, we allow neural networks to directly use the depth fusion prior with the inferred occupancy as the learned implicit function. Our attention mechanism works with either a one-time fused TSDF that represents a whole scene or an incrementally fused TSDF that represents a partial scene in the context of Simultaneous Localization and Mapping (SLAM). Our evaluations on widely used benchmarks including synthetic and real-world scans show our superiority over the latest neural implicit methods.

widely used Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors
Pengchong Hu
Zhizhong Han
Machine Perception Lab, Wayne State University, Detroit, USA pchu@vayne.cm edu
h312h0wayne.
Abstract
Learning neural implicit representations has achieved remarkable performance in 3D reconstruction from multi-view images. Current methods use volume ren- dering render implicit representations into either RGB depth images that are supervised by multi-view ground truth. However, rendering a view each time suffers from incomplete depth at holes and unawareness of occluded structures from the depth supervision, which severely affects the accuracy of geometry in- ference via volume rendering. To resolve this issue, propose to learn neural implicit representations from multi-view RGBD images through volume rendering with an attentive depth fusion prior. Our prior allows neural networks per- ceive coarse 3D structures from the Truncated Signed Distance Function (TSDF) fused from all depth images available for rendering. The TSDF enables access- ing the missing depth at holes on one depth image and the occluded parts that are invisible from the current view. By introducing novel attention mecha- nism, we allow neural networks to directly use the depth fusion prior with the inferred occupancy as the learned implicit function. Our attention mechanism works with either one-time fused TSDF that represents whole scene incrementally fused TSDF that represents a partial scene in the context of Si- multaneous Localization and Mapping (SLAM). Our evaluations benchmarks including synthetic and real-world scans show our superiority over the latest neural implicit methods. Please wt//Ateentive_DD. oF_Prior/ htttsss//machhnpppccppiooooll github io
project page for code and data at
Introduction
3D reconstruction from multi-vien images has been studied for decades. Traditional methods like Structure from Motion (SfM) [61], Multi-View Stereo (MVS) [62]. and Simultaneous Localization and Mapping (SLAM) [49] estimate 3D structures as point clouds by maximizing multi-view color consistency. Current methods [84 28 mainly adopt data-driven strategies learn depth estimation priors from large scale benchmarks using deep learning models. However, the reconstructed 3D point clouds lack geometry details, due to their discrete representation essence, which makes them not friendly to downstream applications.
More recent methods use implicit functions such as signed distance functions (SDFs) 777 occu- pancy functions [51] as continuous representations of 3D shapes and scenes. Using volume rendering. can learn neural implicit functions by comparing their 2D renderings with multi-view ground truth including color 88 197] depth
[8] 3 97 or normal 88 76 16 maps. Although the supervision of
using depth images rendering target can provide detailed structure information and guide impor- tance sampling along rays 88). both the missing depth at holes and the unawareness of occluded structures make it hard to significantly improve the reconstruction accuracy. Hence, how more
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

effectively leverage depth supervision for geometry inference through volume rendering is still challenge.
To overcome this challenge, we introduce to learn neural implicit through volume rendering with an attentive depth fusion prior. Our key idea is to provide neural networks the flexibility of choosing geometric clues, i.c., the geometry that has been learned and the Truncated Signed Distance Function (TSDF) fused from all available depth images, and combining them into neural implicit for volume rendering. We regard the TSDF as a prior sense of the scene, and enable neural networks to directly use it as a more accurate representation. The TSDF enables accessing the missing depth holes on one depth image and the occluded structures that are invisible from the current view, which remedies the demerits of using depth ground truth to supervise rendering. To achieve this, we introduce an attention mechanism to allow neural networks to balance the contributions of currently learned geometry and the TSDF the neural implicit, which leads the TSDF into an attentive depth fusion prior. Our method works with either known camera poses or camera tracking in the context of SLAMM, where our prior could be either a one-time fused TSDF that represents whole scene or incrementally fused TSDF that represents a partial scene. We evaluate our performance on benchmarks containing synthetic and real-world scans, and report our superiority over the latest methods with known estimated camera poses. Our contributions are listed below.
i) We present novel volume rendering framework to learn neural implicit representations from RGBD images. We enable neural networks to use either currently learned geometry or the one from depth fusion in volume rendering, which leads to a novel attentive depth fusion prior for learning neural implicit functions inheriting the merits of both the TSDF and the inference.
ii) We introduce a novel attention mechanism and a neural network architecture to learn attention weights for the attentive depth fusion prior in neural rendering with either known camera poses or camera pose tracking in SLAM.
iii) We report the state-of-the-art performance in surface reconstruction and camera tracking on benchmarks containing synthetic and real-world scans.
2 Related Work
Learning 3D implicit functions using neural networks has made huge progress 661
62,
46
51
88
78
73
76 16
[56]
34
25
11
93
42
44
[ [4
4 (3 5 [9 26 47 20 70
12
48
555
17
27
79
35
32
21
7
58
86
52
We can learn neural implicit functions from 3D ground truth
23
54
45
68
699. 3D point clouds [94] [1 15 2 I0 or multi-view images (46 14
78
73
76
We briefly review methods with multi view supervision below.
2.1 Multi-view Stereo
employs multi-view photo consistency to estimate depth maps from multiple RGB images. They rely on matching key points on different views, and are limited by large viewpoint variations and complex illumination. Without color, space carving [30] is also an effective way of reconstructing 3D structure as voxel grids.
Classic multi-view stereo (MVS) [61 62
Recent methods employ data driven strategy to train neural networks to predict depth maps from either depth supervision 84 or multi-view photo consistency 995
2.2 Neural Implicit from Multi-view Supervision
Early works leverage various differentiable renderers 63 learned implicit functions into images, so that we can measure the error between rendered images and ground truth images. These methods usually require masks to highlight the object, and use surface
38, 26 89
37
30
50
36
67 to render the
rendering to infer the geometry, which limits their applications in scenes. Similarly, DVR [50) and
IDR [87] predict the radiance near surfaces.
With volume rendering, NeRF [46] and its variations 153
60
75
model
geometry and color together. They can generate plausible images from novel viewpoints, and do not need masks during rendering procedure. UNISURF [5T) and NeuS [77] learn occupancy functions
and SDFs by rendering them with colors using revised rendering equations. Following methods

---

TSDF Interpolation O
Within
Koluma Kendering
Bandes
Bundwrath
Rendered RGB fj
Outside
Dandwidth
84
RGB Image
K
Depth T Fussio
TSDF Gr
Rendered Depth Dj
Xnown or Estimated Camera Pose M
Loumablc Feature Grids
Trilinesr
Feature Interpolation
Available Depth Images (DA)
Figure Overview of our method.
improve accuracy of implicit functions using additional priors or losses related depth
g8
normals 88 76 16,, and multi-view consistency [14.
Depth images are also helpful to improve the inference accuracy, Depth information can guide sampling along rays [88] or provide rendering supervision 88 neural networks estimate surfaces.
96
98 which helps
2.3 Neural Implicit with SLAM
Given RGBD images, more recent methods 91
employ neural implicit representations in SLAM. iMAP [66] shows that an MLP serve the only scene representation in a realtime SLAM system. NICE-SLAM (97] introduces a hierarchical scene representation to reconstruct large scenes with more details. NICER-SLAM [96] uses easy-to-obtain monocular geometric cues without requiring depth supervision. Co-SLAM 774 jointly uses coordinate and sparse parametric encodings to learn neural implicit functions. Segmentation priors (29) [8] show their potentials to improve the performance of SLAM systems. With segmentation priors, vMAP 29 represents each object in the scene as neural implicit in a SLAM system.
81
72 59
Instead of using depth ground truth to only supervise rendering, which contains incomplete depth and unawareness of occlusion, allow neural network directly use depth fusion priors part of neural implicit, and determine where and how to use depth priors along with learned geometry.
3 Method
Overview. We present an overview of our method in Fig. Given RGBD images {4, / from view angles, where I and D, denote the RGB and depli images, available either all together or in a streaming way, we aim infer the geometry of the scene as an occupancy function which predicts the occupancy f(q) at arbitrary locations that are either known or estimated by our camera tracking method in the context of SLAM.
(z, 2). Our method works with camera poses (M,}
We learn the occupancy function through volume rendering using the RGBD images (7,, D2) as supervision. We render f with a color function which predicts an RGB color c(q) at locations into RGB image I and depth image Dj. which are optimized minimize their rendering errors the supervision (T, D-
We start from shooting rays {V*} from current view L,, and sample queries q along each ray Vg. For each query q. employ Iearnable feature grids Gr. Gh. and G. covering the scene to interpolate its hicrarchical geometry features t, th, and a color feature t. by trilinear interpolation. Each feature is further transformed into occupancy probability
RGB color by their corresponding decoders fi, fu. and fc. For queries inside the bandwidth of TSDF grid G, fused from available depth images {D.). we leverage its interpolation from G, prior of coarse occupancy estimation. The prior is attentive by neural function fa which determines the occupancy function f(q) by combining currently learned geometry and the coarse estimation using the learned attention weights from Sa-
Lastly, we use the occupancy function f()) and the color function e(q) to render and D' through volume rendering. With learned f. we run the marching cubes [0) to reconstruct a surface.
3

---

Depth Fusion and Bandwidth Awareness. With known or estimated camera poses {M,}, we get TSDF by fusing depth images } that are available either all together or in streaming way. Our method can work with TSDF that describes either whole scene or just part of the scene, and we will report the performance with different settings in ablation studies. We use the TSDF to provide a coarse occupancy estimation which removes the limit of the missing depth or the unawareness of occlusion on single depth supervision. The TSDF is grid which predicts signed distances at any locations inside it through trilinear interpolation. The predicted signed distances are truncated with threshold μ into a range of [-1,1] 1] which covers a band area on both sides of the surface in Fig. |
Since the TSDF predicts signed distances for queries within the bandwidth with higher confidence than the ones outside the bandwidth, we only use the depth fusion prior within the bandwidth. This leads to different ways of learning geometries, as shown in Fig. [ the bandwidth, we model the low-frequency surfaces using low resolution feature grid G), learn the high-frequency details as a complementation using a high resolution feature grid Gr, and let our attention mechanism to determine how to use the depth fusion prior from the TSDF G Instead, we only use the low resolution feature grid G1 to model densities outside the bandwidth during volume rendering. Regarding color modeling, we use the feature grid G2 with the same size for interpolating color of queries over the scene.
Specifically, for queries within
Feature Interpolation. For queries q sampled along rays, we use the trilinear interpolation to obtain its features h,, fug, and he from learnable Gt, G₁, Ge and occupancy estimation o, from G.. Both feature grids and the TSDF grid cover the whole scene and use different resolutions, where Iearnable feature vectors are associated with vertices on each feature grid. Signed distances at vertices on G, may change if we incrementally fuse depth images in the context of SLAMM
Similar to NICE-SLAM 197], we use pre-trained decoders fi and fr to predict low frequency occupancies or and high frequency ones on from the interpolated features & and b, respectively. We use fi and fn as an MLP decoder in ConvONet (54) respectively, and minimize the binary cross-entropy loss to fit the ground truth. After pre-training, we fix the parameters in ft and fh. and use them to predict occupancies below,
Occupancy Prediction Priors.
or = fi(g, ti); on [q, (n), Oth = 01 + Oh
where we enhance fr by concatenating fi as fr. + [fn, fil and denote on as the occupancy predicted at query q by the learned geometry.
Color Predictions. Similarly, we use the interpolated feature te and an MLP decoder fe to predict
color query q, i.e., c(q) fe(g, to). We predict color to render RGB images for geometry inference or camera tracking in SLAM. The decoder is parameterized by parameters 0 which are optimized with other learnable parameters in the feature grids.
We introduce an attention mechanism to leverage the depth fusion prior. We use a deep neural network to learn attention weights to allow networks to determine how to use the prior in volume rendering.
Attentive Depth Fusion Prior.
As shown in Fig. we interpolate signed distance E [- at query q from the TSDF G. using trilinear interpolation, where G, is fused from available depth images. We formulate this interpolation as = (q) E []. We normalize s into an occupancy o, E [0, 1), and regard o, as the occupancy predicted at query q by the depth fusion prior.
For query within the bandwidth, our attention mechanism trains an MLP fa to learn attention weights a and 3 to aggregate the occupancy Oth predicted by the learned geometry and the occupancy o, predicted by the depth fusion prior, which leads the TSDF to an attentive depth fusion prior. Hence, our attention mechanism can be formulated as,
[a, 3] = Sa(ou. and a + =1
We implement fo using an MLP with layers. The reason why we do not use a complex network like Transformer is that we want to justify the effectiveness of our idea without taking too much credit from complex neural networks. Regarding the design, we not use coordinates or positional encoding as part of input to avoid noisy artifacts. Moreover, we leverage Softmax normalization layer to achieve a + 3 = 1, and we do not predict only one parameter a and use 1 a as the second weight, which also degenerates the performance. We will justify these alternatives in experiments.
4

---

For query q outside the bandwidth, we predict the occupancy using the feature t; interpolated from the low frequency grid G2 and the decoder fi to describe the relatively simpler geometry. In summary, we eventually formulate our occupancy function as piecewise function below,
a ou + 3 Os. 01,
(q (-1,1)
f(g)
(g 1 or -1
Volume Rendering. We render the color function e and occupancy function f into RGB I' and depth D' images to compare with the RGBD supervision (T, D).
With camera poses Mj, shoot a ray Vi from view I5. V& starts from the camera origin m and points a direction of r. We sample N points along the ray Vi, using stratified sampling and uniformly sampling near the depth, where each point is sampled qa m dar and dn corresponds to the depth value of qn on the ray. Following UNISURF [51], we transform occupancies I( into weights wo, which is used for color and depth accumulation along the ray V& in volume rendering.
n- W/ ((ga) II (1 I(gn)), 1(ky
Σ Wh x c(gw), D(kY
Σ Why x dre-
LOss. With known camera pose Mj, we render the scene into the color and depth images randomly sampled K pixels on the j-th view, and optimize parameters by minimizing the rendering errors,
J.K
L1=
1
Σ jkk=1
LD
Σ [D)(()) 3A-11
min G.G.G.G.G.
LD+ (5)
JK
JK
In the Context of SLAM. We can jointly do camera tracking and learning neural implicit from streaming RGBD images. To achieve this, we regard the camera extrinsic matrix M, as learnable parameters, and optimize them by minimizing our rendering errors. Here, we follow [97] to weight the depth rendering loss to decrease the importance at pixels with large depth variance along the ray,
J.K
min M,
Vor(D)(k)) [D2(k) - D((1)) + AAL J,k=1
Σ
(6)
JK
where (ar(D)(k)) = > Wn (D) d. Moreover, with streaming RGBD images, incrementally the most current depth image into TSDF G,. Specifically, the incremental fusion includes a pre-fusion and an after-fusion stage. The pre-fusion aims to use a camera pose coarsely estimated by a traditional method to fuse a depth image onto a current TSDF to calculate a rendering error for more accurate pose estimation at current frame. The after-fusion stage will refuse the depth image onto the current TSDF for camera tracking at the next frame. Please refer to our supplementary materials for more details.
We do tracking and mapping iteratively. For mapping procedure, we render E frames each time and back propagate rendering errors to update parameters. E frames include the current frame and E - 1 key frames that have overlaps with the current frame. For simplicity, we merely maintain a key frame list by adding incoming frames with an interval of 50 frames for fair comparisons.
Details. The optimization is performed at three stages, which makes optimization converge better. We first minimize Lp by optimizing low frequency feature grid G and then both low and high frequency feature grid G and Gr, and finally minimize E 35yy optimizing G1, Gh, and G Our bandwidth from the TSDF G, covers 5 voxels on both sides of the surface. We shoot K = 1000 or 5000 rays for reconstruction or tracking from each view, and render fair comparison with other methods. We set A = 0.2, A = 0.5 in loss functions. We sample N = 48 points along each ray for rendering. More implementation details can be found in our supplementary materials.
5 or 10 frames each time for
4 Experiments and Analysis
4.1 Experimental Setup
Datasets. We report evaluations on both synthetic datasets and real scans including Replica (64 and ScanNet [13]. For fair comparisons, we report results on the same scenes from Replica and ScanNet
5

---

Table Reconstruction Comparisons on Replica.
COLMAP (62
TSDF [0]
iMAP [66
22
NICE (2]
Vos [3
DROID 711
NICER X6
Ours
.MAP 70
Ours
DepthL1
3.53
Ace: 5
Comp.
12.12
3.33
5.33
10.19
3.00
4.53
12.29
4,16
2.45
2.39
2.28
Ratio
1
67.62
87.61
66.60
72.96
89.33
86.59
63.62
79.37
92.79
92.99
93.38
as the latest methods, Specifically, we report comparisons on all 8 scenes in Replica. As for ScanNet, we report comparison scene 50, 84, 580, and 616 used by MonoSDF [88]. and also scene 59, 106, 169, and 207 used by NICE-SLAM [97]. We mainly report average results over the dataset, please refer our supplementary materials for results on each scene.
Metrics. With the learned occupancy function f. we reconstruct the surface scene by extracting the zero level set of using the marching cubes algorithm [40]. Following previous studies 197 22].
we use depth I.1 [em], accuracy [cm], completion [cm], and completion ratio [<5em%] as metrics evaluate reconstruction accuracy
report Chamfer distance (LII, precision, recall, and F-score with a threshold of 0.05 to evaluate reconstruction accuracy on ScanNet. To evaluate the accuracy in camera tracking, we use ATE RMSE [65] metric. We follow the
Replica. Additionally,
same parameter setting in these metrics as 997]
4.2 Evaluations
Evaluations on Replica. We use the same set of RGBD images as [66 96 We report evaluations in surface reconstruction and camera tracking in Tab. I Tab. and Tab. respectively.
We jointly optimize camera poses and learn ge- ometry in the context of SLAM. The depth fu- sion prior Gs incrementally fuses depth images using estimated camera poses. We report the accuracy of reconstruction from G2 as "TSDF". Compared to this baseline, we can see that our method improves the reconstruction using the geometry learned through volume rendering and occupancy prediction priors. We visualize the advantage of learning geometry in Fig. 2 Note that the holes in TSDF are caused by the absence of RGBD scans. We plicit keeps the correct structure in TSDF and plausibly completes the missing structures in TSDF We further visualize the attention weights learned for depth fusion in Fig. 3 We visu- alize the cross sections on scenes, where | learned bandwidth are shown color red. Generally, the neural implicit mostly focuses
TSDF
Ours
that neural im-
Figure 2: Merits of attentive depth fusion prior: more on the depth fusion priors in areas where TSDF is complete, while focusing more on the learned geometry in areas where TSDF is incomplete. In the area where TSDF is complete, the network also pays some attention to the inferred occupancy because the occupancy interpolated from TSDF may not be accurate, especially on the most front of surfaces in Fig.
Moreover, our method outperforms the latest implicit-basee SLAM meth- 997 and ods like NICE-SLAM
66
NICER-SIAM [96). We present vi- sual comparisons in Fig. 6 where our method produces more accu- rate and compact geometry. For method using GT camera poses like vMAP 29 and MonoSDF [88) we achieve much better performance as shown by "Ours*" Tab. aad "Ours" in Tab. 2 Although we require the ab- solute dense depth maps, compared to MonoSDF [88] that can work with
Figure 3: Visualization attention on depth fusion.

---

Table 2: Reconstruction Comparison with MonoSDF on Replica.
Test Split
Train Split Chamfer-
Normal C.f
Chamfer- L1 +
F-score t
Normal C.t
F-score
MonoSDF
88
90.56
4.26
76.42
91.80
3.59
85.67
Ours
90.69
2.43
92.47
91.05
2.73
90.52
Table 3: Camera Tracking Comparisons (ATE RMSE) on Replica.
rm-0
rm-1
mm-
off-0
off-1
off-2
off-3
off-4
Avg.
NICE-SLAM 97
1.69
2.04
1.55
0.99
0.90
1.39
3.97
3.08
1.95
NICER-SLAM
96
1.36
1.60
1.14
2.12
3.23
2.12
1.42
2.01
1.88
Ours
1.39
1.55
2.60
1.09
1.23
1.61
3.61
1.42
1.81
scaled depth maps, method only MonoSDF 88 can utilize all views during the whole training process.
the views in front of the current time step, where
In camera tracking, our results in Tab.3 achieve the best in average. We compare the estimated trajectories on
Scaee SeaaaaaNe
Sun ldlt SSadiin
(Replic)
scene off-4 in Fig. (c). The compar-
ison shows that our attentive depth fu-
sion prior can also improve the camera tracking performance through volume rendering.
Evaluations on ScanNet. We further evaluate our method on ScanNet. For surface reconstruction, we compare with the latest methods for learning neural implicit from multiview images. We report both their results and ours with GT camera poses, where we also fuse every depth images into the TSDF G, and render every 10 frames for fair comparisons. Numerical comparisons in Tab, show that our method achieves the best performance in terms of all metrics, where we use the culling strategy introduced MonoSDF [88] to clean the reconstructed mesh. We highlight our significant improvements in visual comparisons in Fig. 7 We see that our method can reconstruct sharper, more compact and detailed surfaces than other methods. We detail comparisons every
Figure 4:
Visual comparisons in camera tracking.
with the top results reported by GO-Surf 775 and NICE-SLAM 997 in Slnd Tab. 6 The comparisons Tab. 5 show that our method achieves higher reconstruction accuracy while GO-Surf produces more complete surfaces. We present visual comparisons with error maps in Fig.
In camera tracking, compare with the latest methods. We incrementally fuse the most current depth frames into TSDF G, which is used for attentive depth fu- sion priors in volume rendering. Numer- ical comparisons in Tab. 7 show that our results are better 2 out of 4 scenes and achieve the best in average. Track- ing trajectory comparisons in Fig. (a)
NICESLAM
G0-Suf
and (b) also show our superiority.
Figure 5: Visual comparison of Error maps (Red: Large).
Roces:0
SLAM
Ours
OB:
NICE-SLAM
Ours
Figure Visual comparisons of error maps (Red: Large) surface reconstructions on Replica.
7

---

Neus
ManhattanSDF
MonoSDF
Neuris
Ours
GT
Figure 7: Visual comparisons in surface reconstructions on ScanNet.
Table 4:
Reconstruction Comparisons on ScanNet.
Acc 4
Comp 4
Chamfer- L
Prec
Recall t
F-score 1
COLMAP 62
0.047
0,235
0.141
0.711
0.441
0.537
UNISURF 55
0.554
0.164
0.359
0.212
0.362
0.267
NeuS 777
0.179
0.208
0.194
0.313
0.275
0.291
VoISDF 85
0.414
0.120
0.267
0.321
0.394
0.346
Manhattan-SDDF 16
0.072
0.068
0.070
0.621
0.586
0.602
NeuRIS 76
0.050
0,049
0.050
0.717
0.669
0.692
MonoSDF 88
0.035
0.048
0.042
0.799
0.681
0.733
Ours
0.034
0.039
0.037
0.913
0.894
0.902
Table 5: Reconstruction Comparison with GO-Surf on ScanNet.
GO-Surf 75
Ours
Scene ID
0050
0084
0580
0616
Avg.
0050
0084
0580
Avg.
Acc |
0.056
0.073
0.057
0.026
0.053
0.030
0.039
0.041
0.026
0.034
Comp
0.024
0.017
0.024
0.023
0.022
0.043
0.014
0.035
0.063
0.039
Chamfer-L L,
0.040
0.045
0,040
0.025
0.038
0.037
0.026
0.038
0.045
0.037
Table 6: Reconstruction Comparison with NICE-SLAM on ScanNet.
NICE 997
Ours
Scene ID
0050
0084
0580
0616
Avg.
0050
0084
0580
0616
Avg.
Acc |
0.030
0.031
0.032
0.026
0.030
0.030
0.039
0.041
0.026
0.034
Comp
0.053
0.020
0.031
0.076
0.045
0.043
0.014
0.035
0.063
0.039
Chamfer-Ly +
0.041
0.025
0.032
0.051
0.037
0.037
0.026
0.038
0.045
0.037
4.3
Ablation Studies
We report ablation studies to justify the effectiveness of modules in our method on Replica. We estimated camera poses in ablation studies.
Effect of Depth Fusion. We first explore the effeet of different ways of depth fusion the performance. According to whether we use GT camera poses or incremental fusion, we try alternatives with G, obtained by fusing all depth images at the very beginning using GT camera poses (Offline) or fusing the current depth image incrementally (Online), and using tracking to estimate camera poses (Tracking) GT' camera poses (GT) in Tab. X The comparisons show that method can work well with either GT or estimated camera poses and fusing depth either all together or incrementally in streaming way. Additional conclusion includes that GT camera poses in either depth fusion and rendering do improve the reconstruction accuracy, and the structure fused from more recent frames is more important than the whole structure fused from all depth images.

---

Table 7: Camera Tracking Comparisons (ATE RMSE) on ScanNet.
Scene ID
0059
0106
0169
0207
Avg.
iMAP 666
32.06
17.50
70.51
11.91
33.00
DI [22
128.00
18.50
75.80
100.19
80.62
NICE 97
12.25
8.09
10.28
5.59
9.05
CO
74
12.29
9.57
6.62
7.13
8.90
Ours
10.50
7.48
9.31
5.67
8.24
Table Ablation Studies on Depth Fusion.
TSDF
Offiine+GT
Online+GT
Offline+Tracking Online+Tracking (Ours)
DepthL1
6.56
2.79
2.73
2.75
3,01
Acc. J
1.56
2.62
2.62
2.82
2.77
Comp. +
3.33
2.45
2.36
2.40
2.45
Ratio f
87.61
92.78
92.79
92.97
92.79
Effeet of Attention. We explore the effect of attention on the performance Tab. First of all, remove the attention mechanism, and observe a severe degeneration in accuracy, which indicates that the attention plays big role in directly applying depth fusion priors in neural implicit. Then, try to apply attention of depth fusion priors geometries predicted by different features including low frequency 4,, high frequency 4n and both of them. The comparisons show that adding details from depth fusion priors onto low frequency surfaces does not improve the performance. More analysis on attention mechanism can be found Fig. 9
Table 9; Ablation Studies on Attention.
w/o Attention
Low
Low High
High (Ours)
DepthL1 Acc.
1.86
2.75
2.96
1.44
2.69
2.96
3,85
2.54
Comp. 1
2.81
3.14
2.91
2.41
Ratio
91.46
89.73
93.63
93.22
Attention Alternatives. Our preliminary re- sults show that using softmax normalization layer achieves better performance than other alternatives. Since we merely need two atten- tion weights and B to combine the learned geometry and the depth fusion prior, one alter- native is one attention weight and use its difference to as another attention weight. However, sigmoid can not effectively take advantage of the depth fusion prior. We also try to coordinates input, which pushes the network to learn spatial sensitive attentions. While the reconstructed surfaces turn out to be noisy in Fig. This indicates that our attention learned general attentive pattern for all locations in the scene. We also report the numerical comparisons in Tab. 10 Please refer to our supplementary materials for more results related to attention alternatives.
Table 10: Ablation Studies on Attention Alternatives.
Coordinates
Sigmoid
Softmax (Ours)
DepthL1 +
1.81
1.96
1.44
Acc. 1
2.66
2.86
2.54
Comp.
2.74
2.66
2.41
sigmoid function predict Ratio 1
93.13
93.27
93.22
No Attention
Sigmoid
Coordinates
Ours
Figure 8: Visual comparison of error maps with different attention alternatives (Red: Large).
Effect of Bandwidth. We further study the effect of bandwidth on the performance in Tab. We try to use attentive depth fusion priors everywhere in the scene with no bandwidth. The degenerated results indicate that the truncated area outside the bandwidth does not provide useful structures to improve the performance. Moreover, wider bandwidth may cause more artifacts caused by the calculation of the TSDF G, while narrower bandwidth brings more incomplete structures. neither of which improves the performance. Instead of using low frequency geometry outside the bandwidth, we also try to use high frequency surfaces. We can see that low frequency geometry is more suitable to describe the scene outside the bandwidth.

---

Table 11: Ablation Studies on the Effect of Bandwidth.
w/o Bandwidth
Bandwidth=3
Bandwidth=5 (Ours)
Bandwidth=7
High
Low (Ours)
Depth1.1 4
1.87
1.61
2.01
124.21
1.44
Acc.
2.40
2.85
2.54
2.93
25.06
2.54
Comp.
3.49
2.61
2.41
2.73
3.62
2.41
Ratio 1
90.94
92.84
93.22
93.55
86.92
93.22
Efffect of Prediction Priors. Prediction priors from decoders fi and fh. also important for accuracy improvements. Instead of using fixed parameters in these decoders,
Table 12: Ablation Studies on Prediction Priors.
w/o Fix
Fix
Fix fu
Fix fr (Ours)
DepthL1
125.95
43.46
F
1.44
try train fi (Fix f1i) or fn (Fix fi) with other parameters. Numerical comparisons in Tab. 12 show that training fi or fn cannot improve the performance, or even fails in camera tracking (Fix fw).
Acc. 1
127.62
67.83
F
2.54
Comp.
56.73
18.79
F
2,41
Ratio 7
5.27
103.51
F
93.22
More Analysis on How Attention Works. Additionally, we do visual analysis on how attention
works Fig. 2 We sample points on the GT mesh, and get the attention weights in Fig. (a). At each point, show its distance to the mesh from the TSDF in Fig. (b) and the mesh from the inferred occupancy Fig. (c), respectively. Fig. (d) indicates where the former smaller than the latter. The high correlation between Fig. focuses more on the occupancy producing smaller errors to the GT surface. Instead, the red in Fig. (e) indicates where the interpolated occupancy is larger than the inferred occupancy is not correlated to the one in (a). The uncorrelation indicates that the attention network does not always focus on the larger occupancy input but the with smaller errors, without reconstructing surfaces.
and Fig. (d) indicates that the attention network
TSDF
fron TSDF
smalle
intepesuael
Figure 9: Analysis Attentions. (a) Map the attention weights occupancies interpolated from TSDF. (b) The point-to-surface distances to the mesh reconstructed from TSDE (c) The point-to- surface distances to the mesh reconstructed from the inferred occupancies. (d) Binary map indicating smaller emoos (red) of TSDE. (e) Binary map indicating larger (red) interpolated occupancies than the inferred occupancies.
5
Conclusion
We propose to learn neural implicit through volume rendering with attentive depth fusion priors. Our novel prior alleviates the incomplete depth holes and the unawareness of occluded structures when using depth images as supervision volume rendering. We also effectively enable neural networks determine how much depth fusion prior be directly used in the neural implicit. Our method can work well with depth fusion from either all depth images together or the ones available in streaming way, using either known estimated camera poses. To this end, our novel attention successfully learns how to combine the learned geometry and the depth fusion prior into the neural implicit for more accurate geometry representations. The ablation studies justify the effectiveness of our modules and training strategy. Our experiments on benchmarks with synthetic and real scans show that our method learns more accurate geometry and camera poses than the latest neural implicit methods.
Acknowledgements and Disclosure of Funding
This work was partially supported by Richard Barber Interdisciplinary Research Award.
10

---

References
[1] Matan Atzmon and Yaron Lipman. SAL: Sign agnostic learning of shapes from raw data. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[2] Matan Atzmon and yaron Lipman. SALD: sign agnostic learning with derivatives. International Conference on Learning Representations, 2021.
[3] Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. JEEE Conference on Computer Vision and Pattern Recognition, pages 6290 139006301,200 2022.
Ma Baorui, Liu Yu-Shen, and Han Zhizhong. Reconstructing surfaces for sparse point clouds with on- surface priors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[5] Ma Baorui, Han Zhizhong, Liu Yu-Shen, and Zwicker Matthias. Neural-pull: Learning signed distance functions from point clouds by learning to pull space onto surfaces. In International Conference on Machine Learning (ICML), 2021.
[6] Aljaz Bozic, Pablo Palafox. Justus Thies, Angela Dai, and Matthias Niesnner Transfommerfuioon: Monoc- ular rgb scene reconstruction using transformers. Advances in Neural Information Processing Systems.
2021.
[7] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing
instructions. CVPR, 2023.
[8] Rohan Chabra, Jan Eric Lenssen, Eddy llg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard A. Newcombe. Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In European Conference on Computer Vision, volume 12374, pages 608-625, 2020.
[9] Chao Chen, Zhizhong Han, and Yu-Shen Liu. Unsupervised inference of signed distance functions from single sparse point clouds without learning priors. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[10] Chao Chen, Yu-Shen Liu, and Zhizhong Han. Latent partition implicit with surface codes for 3d represen- tation. In European Conference on Computer Vision, 2022.
[11] Chao Chen, Yu-Shen Liu, and Zhizhong Han. Gridpull: Towards scalability in learning implicit represen- tations from 3d point clouds. In Proceedings of the IEFE International Conference on Computer Vision (ICCV), 2023.
[12] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In The
Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[13] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. CoRR, abs/1 702.04405, 2017.
[14] Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. Geo-Neus: Geometry-consistee neural implicit surfaces learning for multi-view reconstruction. In Advances in Neural Information Processing Systems, 2022.
[15] Amos Gropp. Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization learning shapes. International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3789-3799, 2020.
[16] Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang. Guofeng Zhang. Hujun Bao, and Xiaowei Zhou. Neural 3d scene reconstruction with the manhattan-wont assumption. JEEE Conference on Computer Vision and Pattern Recognition, 2022.
[17] Anchit Gupta, Wenhan Xiong. Yixin Nie, Ian Jones, and Barlas Oğuz. 3dgen: Triplane latent difffusion for textured mesh generation, 2023.
[18] Yasaman Haghighi. Suryansh Kumar, Jean- Philippe Thiran, and Luc Van Gool. Neural implicit dense semantic slam, 2023.
[19] Zhizhong Han, Guanhui Qiao, Yu-Shen Liu, and Matthias Zwicker. Seqxy2seqz: Structure learning for 3d shapes by sequentially predicting Id occupancy segments from 2d coordinates, 2020.
[20] Ayaan Haque, Matthew Tancik. Alexei A. Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerf2nerf: Editing 3d scenes with instructions, 2023.
[21] Amir Hertz, Ron Mokady, Jay Tenenbaum, Ktir Aberman, Yael Pritch, and Daniel Cohen- Or. Prompt-to- prompt image editing with cross attention control. 2022.
[22] Jiahui Huang. Shi-Shen Huang, Haoxuan Song, and Shi-Min Hu. Di-fusion: Online implicit 3d recon- struction with deep priors, 2021.
[23] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießnnrr and Thomas Funkhouser. Local implicit grid representations for 3D scenes. In JEEE Conference on Computer Vision and Pattern Recognition, 2020.
[24] Hanqi Jiang, Cheng Zeng. Runnan Chen, Shuai Liang, Yinhe Han, Yichao Gao, and Conglin Wang. Depth-neus: Neural implicit surfaces learning for multi-vies view reconstruction based on depth information optimization. 2023.
11

---

[25] Sijia Jiang, Jing Hua, and Zhizhong Han. Coordinate quantized neural implicit representations for multi-view 3d reconstruction. In IEEE International Conference on Computer Vision, 2023.
[26] Yue Jiang. Dantong Ji, Zhizhong Han, and Matthias Zwicker. SDFDiff: Diffcrentiable rendering of signed distance fields for 3D shape optimization. In IEEE Conference Computer Vision and Pattern Recognition, 2020.
[27]
Leonid Keselman and Martial Hebert. Flexible Techniques for Differentiable Rendering with 3D Gaussians, 2023.
[28] Lukas Koestler, Nan Yang, Niclas Zeller, and Daniel Cremers. TANDEM:
tracking and dense mapping in real-time using deep multi-view stereo. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Conference on Robot Learning, volume 164, pages 34-45, 2021.
[29] Xin Kong, Shikun Liu. Marwan Taher, and Andrew J Davison. vmap: Vectorised object mapping for neural field slam. arXiv preprint 2r30::2302..m 20..2.1302.2..... 2023.
[30] A. Laurentini. The visual hull concept for silhouette- image understanding. IEEE Transactions on Puttern Analysis and Machine Intelligence. 16(2):150-162 1994.
[31] Seunghwan Lee, Gwanmo Park, Hyewon Son, Jiwon Ryu, and Han Joo Chae. Fastsurf: Fast neural rgb-d surface reconstruction using per-frame intrinsic refinement and tsdf fusion prior learning. arXiv preprint arXiv:2.303.00 035088 2023.
[32] Jiabao Lei, Jiapeng Tang, and Kui Jia. Rgbd2: Generative scene synthesis via incremental view inpainting using rgbd diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
[33] Tianyang Li, Xin Wen, Yu-Shen Liu, Hua Su, and Zhizhong Han. Learning deep implicit functions for 3d shapes with dynamic code clouds. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
(34] Zhaoshuo Li. Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen- Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2023.
[35] Zhihao Liang. Zhangjin Huang, Changxing Ding, and Kui Jia. Helixsurf: A robust and efficient neural implicit surface learning of indoor scenes with iterative intertwined regularization, 2023,
[36] Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey. SDF-SRN: Learning signed distance 3D object
reconstruction from static images. In Advances in Neurul Information Processing Systems, 2020.
[37] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3D supervision. In Advances in Neural Information Processing Systems, 2019.
[38] Shaohui Liu, Yinda Zhang, Songyou Peng. Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. DIST: Rendering deep implicit signed distance function with differentiable sphere tracing. In IEEE Conference on Compuler Vision and Pattern Recognition, 2020.
[39] Shi-Lin Liu, Hao-Xiang Guo, Hao Pan, Pengshuai Wang, Xin Tong, and Yang Liu. Deep implicit moving lcast-squares functions for 3D reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[40] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3D surface construction algorithm. Computer Graphics, 21 21(4):163-169 1987.
[41] Baorui Ma, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Neural-pull: Learning signed distance functions from point clouds by learning to pull space onto surfaces. In International Conference on Machine Learning, 2021.
[42] Baorui Ma, Yu-Shen Liu, and Zhizhong Han. Learning signed distance functions from noisy 3d point clouds via noise to noise mapping. In International Conference on Machine Learning (ICM.), 2023.
[43] Baorui Ma, Yu-Shen Liu, Matthias Zwicker. and Zhizhong Han. Surface reconstruction from point clouds by learning predictive context priors. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[44] Baorui Ma, Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Towards better gradient consistency for neural signed distance functions via level set alignment. In Conference on Computer Vision and Patter Recognition (CVPR), 2023.
[45] Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. ACORN: adaptive coordinate networks for neural scene representation. CoRR, abs/2 00022105..2777 02788,878887 2021.
[46] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren
Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
[47] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. arXiv:2201.05999, 2022.
[48] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8280- 8290, June 2022.
12

---

[49] Richard A. Newcombe. Steven J. Lovegrove, and Andrew J. Davison. Dtam: Dense tracking and mapping in real-time. In International Conference on Computer Vision, pages 2320-2327, 2011.
[50] Michael Niemeyer, Lars Mescheder, Michael Occhslc, and Andreas Geiger. Diffcrentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In IEEE Conference on Computer Vision and Pottern Recognition, 2020.
[51] Michael Oechsle, Songyou Peng. and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In International Conference on Computer Vision, 2021. [52] Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhnefer, and Mustafa Mukadam. isdf: Real-time neural signed distance fields for rohot perception. In Robotics: Science and Systems, 2022.
[53] Keunhong Park, Utkarsh Sinha. Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo
Nerties: Deformable neural radiance fields. IEEE International Conference
on Computer Vision, 2021.
[54] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision, 2020.
[55] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
[56] Radu Alexandru Rosu and Sven Behnke. Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[57] Darius Rückert, Linus Franke, and Mare Stamminger. Adop: Approximate differentiable one-pixel point rendering. wxXi:2210.06633: 2021.
[58]
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023.
[59] Erik Sandström, Kevin Ta, Luc Van Gool, and Martin R. Oswald. Uncle-slam: Uncertainty learning for dense neural slam, 2023.
[60] Sara Fridovich- Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In JEEE Conference on Computer Vision and Pottern Recognition, 2022.
[61]
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-frm--nott revisited. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[62]
Johannes I. Lutz Schönherger, Enliang Zheng, Mare Pollefeys, and Jan-Michacl Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision, 2016.
[63] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous ODDstructure-award neural scene representations. In Advances in Neural Information Processing Systems, 2019.
[64] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales. Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele. Steven Lovegrove, and Richard A. Newcombe. The replica dataset: digital replica of indoor spaces. CoRR. 15s7199 2019.
[65] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Syxtems, pages 573-580, 2012.
[66] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison. imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages
6229-6238, 2021.
[67] Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. NeuralRecon: Real-time coherent 3D reconstruction from monocular video. IEEE Conference on Computer Vision and Puttern Recognition, 2021.
[68] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In IEFE Conference on Computer Vision and Pattern Recognition, 2021.
[69] Jiapeng Tang. Jiabao Lei. Dan Xu, Feiying Ma, Kui Jia, and Lei Zhang. SA-ConvONet: Sign-agnostic optimization of convolutional occupancy networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[70] Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng. Del- icate textured mesh recovery from nerf via adaptive surface refinement. arXiv preprint 1, 2022.
[71] Zachary Teed and Jia Deng. DROID-SLAM Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. Advances in neural information processing systems, 2021.
13

---

[72]
Andreas L. Teigen, Yeonsoo Park, Annette Stahl, and Rudolf Mester. Rgb-d mapping and tracking in plenoxel radiance field, 2023.
[73] Delio Vicini, Sébastien Speierer, and Wenzel Jakoh. Differentiable signed distance function rendering. ACM Transactions Graphics, 41(4):125:11155:18 2022.
[74] Hengyi Wang. Jingwen Wang. and Lourdes Agapito. Co-slam: Joint coordinate and sparse parametric
encodings for neural real-time slam, 2023.
[75] Jingwen Wang. Tymoteusz Bleja, and Lourdes Agapito. Go-surf: Neural feature grid optimization for fast, high-fidelity rgb-d surface reconstruction. In International Conference on 3D Vision, 2022.
[76] Jiepeng Wang. Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, and Wenping Wang. NeuRIS: Neural reconstruction of indoor scenes using normal priors. European Conference on Computer Vision, 2022.
[77] Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advances in Neural Information Processing Systems, pages 77171-27183, 2021.
[78] Yiqun Wang. Ivan Skorokhodov, and Peter Wonka, HF-NeuS: Improved surface reconstruction using high-frequency details. 2022.
[79] Chao Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari. Multiview
compressive coding for 3d reconstruction, 2023.
[80] Yunjie Wu and Zhengxing Sun. DFR: differentiable function rendering for learning 3D generation from
images. Computer Graphics Forum, 39(5):241 252, 2020.
[81] Liu Xinyang, Li Yijin. Teng Yanbin, Bao Hujun, Zhang Guofeng, Zhang Yinda, and Cui Zhaopeng. Multi-modal neural radiance field for monocular dense slam with a light-weight tof sensor, In International Conference on Computer Vision (ICCV), 2023.
[82] Xiangyu Xu, Lichang Chen, Changjiang Cai, Huangying Zhan, Qingan Yan, Pan Ji, Junsong Yuan, Heng Huang, and Yi Xu. Dynamic voxel grid optimization for high-fidelity rgh-d supervised surface reconstruction, 2023.
[83] Xingrui Yang. Hai Li, Hongjia Zhai, Yuhang Ming, Yugian Liu, and Guofeng Zhang. Vox-fusion: Dense tracking and mapping with voxel-based neural implicit representation. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Dec 2022.
[84] Yao Yao, Zixin Luo, Shiwei Li. Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. Europea Conference on Computer Vision, 2018.
[85] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Advances in Neural Information Processing Systems, 2021.
[86] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski. Jonathan T. Barron, and Ben Mildenhall. Bakedsdf: Meshing neural sdfs for real-time view synthesis. arXiv, 2023.
[87] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33, 2020.
[88] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction. ArXiv, abs/ 000666 2022. [89] Sergey Zakharov, Wadim Kehl, Arjun Bhargava, and Adrien Gaidon. Autolabeling 3D objects with differ- entiable rendering of sdf shape priors. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[90] Andy Zeng. Shuran Song, Matthias NieBner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In CVPR, 2017.
[91] Youmin Zhang, Fabio Tosi, Stefano Mattoccia, and Matteo Poggi. Go-slam: Global optimization for consistent 3d instant reconstruction. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2023.
[92] Wenbin Zhao, Jiabao Lei, Yuxin Wen, Jianguo Zhang, and Kui Jia. Sign- implicit of surface self-similarities for shape modeling and reconstruction from raw point clouds. CoRR, 2.0049, 2020.
[93] Junsheng Zhou, Baorui Ma. Shujuan Li, Yu-Shen Liu, and Zhizhong Han. Learning a more continuous zero level set in unsigned distance fields through level set projection. In Proceedings of the IEEE/CVF international conference computer vision, 2023.
[94] Junsheng Zhou, Baorui Ma, Yu-Shen Liu, Yi Fang, and Zhizhong Han. Learning consistency-uware unsigned distance functions progressively from raw point clouds. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[95] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In IEFF Conference on Computer Vision and Pattern Recognition, pages 6612-6619, 2017.
[96] Zihan Zhu, Songyou Peng, Viktor Larsson, Zhaopeng Cui, Martin R. Oswald, Andreas Geiger, and Marc Pollefeys. NICER-SLAM: neural implicit seene encoding for RGB SLAM. CoRR, abs/2302.03594, 2023.
14

---

[97] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In IEEE Conference Computer
Vision and Pattern Recognition, 2022.
[98] Zi-Xin Zou, Shi-Sheng Huang. Yan-Pei Cao. Tai-Jiang Mu, Ying Shan, and Hongbo Fu, Mononeuralfusion: Online monocular neural 3d reconstruction with geometric priors. CoRR, abs/2209.1515... 2022.
15


Reviews:
[{"review_id": "AKMZEL3kP9", "reviewer": "Reviewer_QXhf", "summary": "This paper aims to improve the performance of 3D reconstruction from multi-view RGB-D images. The key innovation is the attentive depth fusion prior, which allows the networks to directly use the depth fusion prior with the inferred occupancy as the learned implicit function. Experiments show that the proposed method work for both one-time fused TSDF and incrementally fused TSDF.", "questions": "1. Since the proposed method is predicting multiple values for the same query points, what is the time increase over a naive neural implicit function (predicting a single signed distance for each query point)?\n\n2. Minor editorial suggestions: \nL173: it would be better to swap \"color function c\" and \"occupancy function f\" for better correspondence.\nL191: includes -> include\nL303: attentions -> attention", "limitations": "The authors have not adequately addressed the limitations or potential negative societal impact. The authors could show some failures cases that indicates some common patterns where the proposed method would fail on", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper aims to improve 3D reconstruction by better handling cases such as incomplete depth at holes. This is one limitation of existing work and is an important task.\n\n2. The method section is clearly and well explained to help readers understand the details.\n\n3. The authors have included video results that can help better understand the visual result quality of the proposed method.\n\n4. Extensive comparison and ablation studies have been done to show improvements over prior work and justify the decision choices of all components.", "weaknesses": "1. L242 claims that the proposed method \"plausibly completes the missing structures from TSDF\" in Fig. 2. However, the completed surface by the proposed method seems to be of very low-quality. It seems a simple non-learning mesh hole filling algorithm on top of the meshes on the left of Fig. 2 can produce much cleaner results. Learning-based methods [A,B] that predict TSDF for indoor scene completion can also achieve much better results. \n\n[A] Dai et al. SG-NN: Sparse generative neural networks for self-supervised scene completion of rgb-d scans. CVPR 2020.\n[B] Dai et al. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. CVPR 2018.\n\n2. It is great that the paper has conducted extensive quantitative experiments. However, there are currently 12 tables in the main paper which seems a bit overwhelming. Indeed, some tables are not that critical and can be moved to supplementary. Alternatively, the authors can only present the scene average evaluation in the main paper for some table and move the per-scene results to the supplementary. The new empty space can be instead used to address some limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Low-quality completion\n\nAs described in L226-228, all completed meshes in the right column in Fig.2 are reconstructed in the context of SLAM. Under this setting, we do camera tracking on each frame, render RGB and depth images every 5 frames for reconstruction by minimizing rendering errors, fuse depth images into TSDF incrementally using the estimated camera poses, and only access 0-th to (t-1)-th GT images at t-th time step. These conditions in SLAM result in a dynamic and incremental context information that is much more incomplete and with more uncertainty than the static and relatively complete context required in scene completion or hole filling algorithms on meshes. This kind of context information determines that our method is only allowed to do an extrapolation-like completion since there is no chance to observe the whole scene. However, data-driven or traditional hole filling methods can access the whole scene and merely do an interpolation-like completion, which makes much difference. As described in L241-242, Fig.2 aims to demonstrate our ability of keeping all the correct structure in TSDF and inferring the missing structure through volume rendering.     \n\nAdditionally, we report comparisons with data-driven or hole filling methods such as SGNN and Filling Holes in Meshes[SGP03] in Fig.8 in the rebuttal. SGNN fails to fill holes in the scene with ceilings, and [SGP03] produces severe artifacts in empty space due to its limited ability of perceiving the context.\n\n2. Layout\n\nWe will follow your suggestions to improve the layout.\n\n3. Time cost on more prediction\n\nActually, we only predict one occupancy at one location, and do not predict multiple ones. Besides the one predicted by our neural network, we interpolated occupancies from the fused TSDF for query points. There is almost no time cost on using trilinear interpolation or the simple attention network. As for the TSDF fusion implemented with CUDA, it merely takes about 25.9s to incrementally fuse all 2000 depth images frame by frame in a scene. For example, in room0 of Replica, the total runtime of our method is 29min56s, which is comparable to 29min19s with NICE-SLAM.\n\n4. Limitations\n\nSince we do not directly model the lighting in the scene, our method does not work well with transparent  surfaces or surfaces with reflection, such as glass. Reflection may make depth sensors fail to capture depth information or break multi-view consistency on RGB images, both of which degenerate the accuracy of reconstructed surfaces. However, this merely makes our method reconstruct a small area poorly but rather fail to reconstruct a whole scene. Please see Fig.3 in the rebuttal for more details.\t\t"}, {"review_id": "VNvFHHfe8T", "reviewer": "Reviewer_X87y", "summary": "The paper proposes a pipeline for estimating scene geometry with TSDF represented with neural implicit function, based on multi-view RGBD inputs. The main novelty is a fusion mechanism which utilized fused depth geometry as prior, and fuses geometry prior with estimated geometry using attention-based weighting. The weighting also considers bandwidth of the TSDF representation, allowing for multi-resolution grid features to be leveraged. The results demonstrated superior performance w.r.t. geometry estimation when compared to methods like MonoSDF, and better pose estimation when compared to methods like NICE-SLAM.", "questions": "There are questions included in my reviews, mostly in Weakness 1.2 and 1.3. it would be great if the authors can address those questions.", "limitations": "Not applicable. No potential negative societal impacts are mentioned.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] The idea to fuse input geometry and estimated geometry, instead of using input geometry as constraints when optimizing estimated geometry, is in general novel and sound. The reasoning is, input geometry from fused depth maps may present high confidence in certain areas, and estimated geometry via MLPs may excel at other regions. By fusing these two instead of estimating all regions with MLPs, the method yields better geometry, and benefits pose estimation in SLAM applications. The insights here would be useful to the community to inspire better ways to leverage input geometry when training MLP-based TSDF fields.\n\n[2] Extensive evaluation. The methods evaluates the method w.r.t. geometry estimation with state-of-the-art methods like MonoSDF, which is the main focus of the paper. Beyond geometry, the method also shows improved geometry benefits downstream tasks of pose estimation, and is able show superior performance over SLAM systems with implicit neural representations. Extensive ablation study is also applaudable, which justifies certain design choices,  as well as providing visualization of what is learned with the attention weights.\n\n[3] The paper is in general well-written and easy to follow.", "weaknesses": "[1] Further clarify and justifications are needed in various parts of the paper. \n\n[1.1] For example, on why Transformers are not used to model the attentions, the paper gives an unconvincing reason that people may conjecture that the performance improvements are mainly contributed to Transformers. However, it is unfair and premature to say so without providing actual results with a Transformer-base design. With those additional results, we may then get a better idea on how important the architecture design of the attention mechanism is to the performance gain. Additionally, design choices mentioned in Line 165-169 require justifications via ablation study.\n\n[1.2] Additionally, on the explainability of the learned attention maps, the paper mentions in Line 246 that, 'some area' are paid more attention to, which is not sufficient. Further analysis onto what part of the input geometry is prioritized, and what part is mostly learned, will give better insights of the attention weights, and better justifies the motivation of the proposed. Without further analysis, the attention machoism behaves more or less as a black box.\n\n[1.3] Finally, additional details of the experiment setting are needed. For example, in comparing against MonoSDF, it is not clear whether input depth or estimated depth (with DPT) are used. In experiments where poses are optimized, it is not clear what the initializations look like, what pose representation and regularization are used, and what was the convergence.\n\n[2] Paper writing. For example, the introduction section repeatedly mentions the core contribution of 'attentive depth fusion priors', without giving any explanation or hints of what it exactly is. As a result, the contribution #2 is essentially explaining the 'attentive depth fusion priors', however it is not immediately clear that the 'attentive depth fusion priors' mentioned in contribution #1 is basically explained in #2. Additionally, more discussion should go into explaining import baselines like MonoSDF and NICE-SLAM, on how are they different from the proposed method, so as to better clarify the contributions, and to provide further insights on why the proposed method performs better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Attention network justification\n\nWe did report ablation studies on attention network justification in our manuscript. We reported numerical and visual comparisons with different attention modeling alternatives in Tab.10 and Fig.8, respectively, where our current attention network produces the best performance. We also reported either the low resolution or high resolution occupancy inferred through volume rendering should be focused on in Tab.11. Additionally, we compared the performance of our simple MLP and the more advanced Transformer network in the rebuttal. Due to the short rebuttal period, we merely saw some results with Transformer that are comparable to ours, which indicates that it still needs more effort to tune the architecture.\n\n2. Attention analysis\n\nWe visualize the learned attention weights on a cross section of the TSDF grid in Fig.3 in the manuscript, and also visualize the attention weights learned at different epochs in our video. Generally, the network mostly focuses more on the occupancy interpolated from the TSDF in areas where TSDF is complete, while focusing more on the occupancy inferred through volume rendering in areas where TSDF is incomplete. In the area where TSDF is complete, the network also pays some attention to the inferred occupancy because the occupancy interpolated from TSDF may not be accurate, especially on the most front of surfaces in Fig.3 in the manuscript. \n\nMoreover, we additionally reported visual analysis on how attention works in Fig.1 in the rebuttal. We sample points on the GT mesh, and get the attention weights in Fig.1(a). At each point, we show its distances to the mesh from the TSDF in Fig.1(b) and the mesh from the inferred occupancies in Fig.1(c), respectively. Fig.1(d) indicates where the former is smaller than the later. The high correlation between Fig.1(a) and Fig.1(d) indicates that the attention network focuses more on the occupancy producing smaller errors to the GT surface. Instead, the red in Fig.1(e) indicating where the interpolated occupancy is larger than the inferred occupancy is not correlated to the one in Fig.1(a). The uncorrelation indicates that the attention network does not always focus on the larger occupancy input.\n\n3. Experimental details\n\nWe are sorry for uploading the wrong supplemental material which includes all the experimental details. Regarding the comparison with MonoSDF, we use the estimated depth maps to report their results. We tried to use GT depth maps and reported their results in Fig.5 and Tab.2 in the rebuttal. But the improvement from GT depth maps is marginal, which is still not as good as ours. Regarding camera tracking, we followed NICE-SLAM and NICER-SLAM to use the GT camera pose for the first frame as the initialization, and also use the same regularization. The camera pose is represented as a 4x4 matrix including the rotation and translation matrices. The optimization is regarded as being converged after a certain amount of iterations in camera tracking and mapping stages respectively. We will upload supplemental materials to report the experimental details.\n\n4. Paper writing\n\nWe will follow your suggestion to revise our manuscript, such as explaining our attentive depth fusion prior and rephrasing our contributions.\n\n5. Discussions on difference to the latest\n\nWe will add more discussions on our advantages over MonoSDF and NICE-SLAM. In summary, our method uses depth fused TSDF as a prior, and employs an attention network to determine where we should use it and how much we should use it with a potential correction from the occupancy inferred through volume rendering. Our attentive depth fusion prior is a more accurate guidance to learn neural implicit representations than using GT depth images as a rendering supervision because single depth images may contain holes and occlusion which can be significantly improved by TSDF. Please see the comparison in Fig.6 in the rebuttal for our advantage over directly using the TSDF as supervision. Moreover, our method also shows novel ways of balancing the confidence within bandwidth and outside bandwidth with attention networks, which also spreads some novel perspectives to use TSDF with neural implicit functions. Our ablation studies in the supplemental material also show that the attentive depth fusion prior can improve surface reconstruction no matter if we are using depth images as rendering targets or not. Hence, the novel way of using depth information differentiates our method from other methods inferring neural implicit representations in either multi-view or SLAM settings, such as MonoSDF and NICE-SLAM.\n"}, {"review_id": "c5idPyX3B5", "reviewer": "Reviewer_3uRZ", "summary": "This paper proposed a simple yet effective approach of neural implicit surface reconstruction from RGB-D seuqences that leverages the reconstructed TSDF grids as a prior which effectively improve the reconstruction quality of fitting a neaural implicit surface from multi-view RGB-D images directly. The idea of using fused TSDF grids as a geometric prior is novel and makes sense as it comes for free from RGB-D sequences. The proposed attention mechnism of fusing the results from TSDF prior and neural implicit function is also novel and effective through experimental results. The authors have also demonstarted the effectiveness of their proposed methods and each of its components through extensive experiments, ablations and analysis. Except from some very minor issues, this is a good paper. For these reasons, I would reccomend \"Weak Accept\" at this stage.", "questions": "Some minor questions:\n\n1. What's the resolution and voxel size used in the feature grids and TSDF grids?\n\n2. Regarding the learnable attention module $f_a(\\cdot)$, seems that it is only conditioned on the two occupancy predictions? Have you tried with more conditions?\n\n3. When evaluating reconstruction quality on ScanNet (Tab. 4), what mesh culling strategy did you use? Also is it consistent across all the methods?\n\n4. In the ablation studies on depth fusion (Tab. 8), why would there be difference between GT All and GT? If I understand correctly, the difference is just that GT All operates in a batched manner while GT is online. But this shouldn't cause difference when you run TSDF-fusion?\n", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The idea of using fused TSDF grids as a prior is novel. Normally, traditional TSDF fusion could achieve better accuracy in geometry reconstruction but falls short of filling the holes in occluded or unobserved regions. Neual implicit methods, on the other hand, is better at hole filling at the cost of slightly less accurate surfaces. Therefore, using TSDF fusion as a prior and combining these two methods is very natual and has very clear motivation. Also as the authors showed, fused TSDF grid already encoded all historical frames and is occlusion-aware, which could also bring benefits to directly fitting to multi-view RGBD images. Moreover, this prior also comes with no extra cost. The proposed learnable attention module to fuse the output from two sources is also a \"simple-yet-effective\" solution.\n\n2. Overall, the quantitative experiments are conducted extensively and are sufficient to show the effectiveness of the proposed methods. Quantitative results show the proposed method achieves SOTA  reconstruction and tracking results compared to previous RGB-D neural implicit SLAM/surface reconstruction methods. Qualitative results also look promising and accompanied with detailed and clear analysis. Ablation studies are also well conducted and justified the effectiveness of each module and design choices.\n\n3. The paper is very well written and easy to follow. The introduction and related work sections are very clear and show clear motivation of the proposed method.\n\n4. The method could run both in an online SLAM manner and under batched offline setting, which increases the system's practical value.", "weaknesses": "1. Overall, the comparisons with RGB-D baselines seems convincing, but some of the experiments in Sec. 4.2 are comparing against baseline methods that rely on RGB-only input. For example in Tab. 2, 4 and Fig. 5. These are not completely fair comparisons as the proposed methods take measured depth directly. Or did you change some settings in the experiments, like supervising the proposed method with predicted depth instead of measured depth?\n\n2. It would be good to also include performance analysis, such as memory and run-time comparison to previous methods, especially for the SLAM setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Experimental settings for comparing with multi-view reconstruction methods\n\nSince our method focuses more on SLAM applications which require both camera tracking and mapping two procedures, it is very hard to conduct completely fair comparisons with multi-view reconstruction methods. The comparison details are shown below.\n\nThe aspect that we take advantage:\n* Using GT depth images. (while RGB only methods do not use them.)\n\nThe aspects that we do not take advantage:\n* Only observe 0-th to (t-1)-th images before the current time step t (while multi-view reconstruction methods can observe all images during the whole training procedure.)\n* Only process frames sequentially (with an interval of 5 frames), and may only see a frame one time during training, if it is not a key frame or far from the current frame that is being processed. (while multi-view reconstruction methods see all images many times during the iterative optimization.)\n\nWe did intend to use the estimated depth images to produce our results. However, we found each estimated depth image used by MonoSDF needs a pair of scale and shift parameters to get normalized, which aligns the estimated point cloud to a sparse point cloud obtained by structure from motion. However, the scale and shift parameters are not consistent across different views, which makes it hard to fuse the estimated depth images into a plausible TSDF, even if using GT camera poses. Fig.7 in the rebuttal shows that the TSDF fails to represent a coarse structure of the scene, which can not be used as a depth fusion prior in our method. However, we tried to use GT depth maps and reported their results in Fig.5 and Tab.2 in the rebuttal. As we see the improvement with GT depth images is marginal, which is still not as good as ours. \n\n2. Computational comparisons\n\nThanks to the TSDF fusion implemented with CUDA, the runtime of our program(29min56s) is comparable to the one with NICE-SLAM(29min19s) for a scene with 2000 frames. It only takes about 26s to fuse depth images frame by frame incrementally. As for the number of parameters, our method contains 12.15M learnable parameters, which is comparable to the 12.20M ones in NICE-SLAM. \n\n3. Resolutions\n\nWe follow NICE-SLAM and some TSDF fusion algorithms to use different resolutions in different scenes while keeping the size of a voxel the same across different scenes. In a scene, the resolutions of feature grids or TSDF grids are determined by the size of the scene and the size of a voxel. Specifically, the size of a voxel in the low resolution feature grid is 0.32, the size of a voxel in the high resolution feature grid is 0.16, and the size of a voxel in the TSDF grid is 1/64. Using the same size of a voxel aims to generalize the geometry prior learned at the same size of voxel grids.\n\n4. Attention network alternatives\n\nAs mentioned in L165-169, we tried different designs. For instance, without using softmax, we use a sigmoid to predict one weight alpha while using 1-alpha as another weight beta, or use coordinates as a condition. We found that the coordinate condition makes the fusion very sensitive to locations, which degenerates performance. We reported these numerical and visual comparisons in Tab.10 and Fig.8 in the manuscript. We also conduct experiments to explore either the low resolution or high resolution occupancy inferred through volume rendering should be attentive in Tab.11 in the manuscript. Moreover, we conduct experiments to report more results with different conditions in Tab.4 and Fig.2 in the rebuttal, and all these alternatives degenerate the reconstruction accuracy.\n\n5. Culling strategy\n\nWe use the culling strategy introduced in MonoSDF to produce our results and the results of other methods on ScanNet.\n\n6. GT All vs GT\n\nThe difference between GT All and GT in Tab.8 in our manuscript lies in if we fuse all depth maps into a TSDF grid at the very beginning. GT All indicates we do that and use the TSDF containing the whole scene to process all frames. GT indicates that we do not do that but incrementally fuse a depth map at the current time step into the TSDF, which means we can only use part of the scene observed before the current time step. Although the TSDF fusion is the same, how the attention network learns weights to balance the occupancy interpolated from the TSDF and the occupancy inferred through volume rendering is different under these two conditions. Thus, the TSDF difference in learning makes a difference in results.\n"}, {"review_id": "3urmHrS88k", "reviewer": "Reviewer_sZZA", "summary": "\nThis paper tackles the problem of 3D scene reconstruction from posed RGB-D images by learning an implicit occupancy function.\n\nUnlike previous methods [58, 3, 65, Ref_DS] that directly use the depth values (obtained either from depth images, SfM pipelines, or from depth prediction networks) of individual rays (i.e. pixels) as supervisory signal leading to learning better scene geometry, faster convergence [Ref_DS, Ref_IN] and inference [Ref_IN, 58], this method proposes to first fuse the depth frames into a TSDF representation using some off-the-shelf method and then query the TSDF for obtaining the occupancy value corresponding to a particular point on a sampled ray for supervision.In other words, the off-the-shelf obtained TSDF of the scene is used as a scene prior.\n\nThe motivation for relying on TSDF representation of the scene is that the depth images obtained from RGBD sensors suffer from missing depth values (or holes) which makes it challenging to supervise 3D point samples corresponding to those rays.\n\nThe primary contribution of this paper is to learn an attention function that, for a 3d point sample,  estimates the weights for the linear combination of occupancy values which are predicted from the neural network and occupancy values obtained from the TSDF scene prior (as in, final_occupancy = w1*predicted_occupancy + w2*TSDF_obtained_occupancy_prior).  In this sense, the TSDF prior becomes an attentive depth fusion prior.\n\nThe network is supervised using volumetric rendering of depth and colour as proposed in [31].\n\nJust like previous methods (e.g. [48, 58]), this method too relies on a hybrid representation: three discrete feature grids are learned, two for low and high-resolution representation of geometry and one for colour, and the tri-linearly interpolated feature value for a corresponding 3D point is decoded by corresponding MLP decoders to obtain occupancy and colour for that 3D point.\n\nThe truncated nature of TSDF (it belongs to (-1 , 1) ) is also exploited to inform when to use low-resolution grid features to get occupancy and when to rely on both low and high-resolution grid features to predict occupancy and fuse it (using learned weights) with the TSDF informed occupancy prior; it is clear that points which have TSDF values under the range of 1 or -1 are near surfaces and hence higher resolution feature grids are necessary and in addition, TSDF will explain such points the best.\n\nThe method is also applicable for SLAM settings where RGB-D frames are obtained incrementally in which the camera extrinsics, i.e. poses, are also optimized with the network parameters.\n\n\n[Ref_IN] Kim, Mijeong, Seonguk Seo, and Bohyung Han. \"Infonerf: Ray entropy minimization for few-shot neural volume rendering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[Ref_DS] Deng, Kangle, et al. \"Depth-supervised nerf: Fewer views and faster training for free.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n", "questions": "1.  How is the proposed attentive depth fusion prior different from simply using a softmax operation over the predicted occupancies and the occupancy obtained from TSDF prior? This question is motivated from the fact that the attention function only takes the two occupancies as input .\n\n2. How is the error in TSDF which is accumulated due to progressive fusion of depth images using their recent estimate of pose handled?\n\nDetail question: In a SLAM setting where we do not have all the RGB-D images beforehand, the TSDF is performed by fusing the scene incrementally using the estimated poses. However, pose estimates from any front-end would have some degree of error. This error would propagate to the TSDF scene prior which is used to construct the final occupancy. The network however is optimized along with camera poses but the TSDF does not seem to be updated based on the optimized camera poses. If, however, it is updated, would it not mean that at every mapping phase, we will have to fuse the entire TSDF again? This would again mean that we will have to keep appropriate number of depth frames for this. However this method, like vMAP[17], iMAP[41], stores keyframes, whose main purpose is for replay while learning incrementally to tackle the catastrophic forgetting issue. No reference to this is provided in the paper.\n\n3. How would the result look like if instead of a linear combination of the predicted occupancy and occupancy from TSDF prior, we use the one with the maximum weight? Could the results indicate that the attention function is redundant? This would also demonstrate if there is any merit in the idea of using TSDF as a prior for composing the occupancies or learning to predict TSDF from TSDF prior is a better idea; [19] has three stages of training, the first is devoted to this.", "limitations": "The authors have not discussed any limitations. However, the following are the two strong limitations of this work:\n\n1. One strong limitation of this work is that it would require the availability of TSDF prior even in test time. So what is the point of learning implicitly? The method is also capable of optimizing the camera poses and hence the resulting occupancy would be better (as also shown in the results). But then how is it different from running a classic bundle adjustment and then fusing the optimized depth values into a more accurate TSDF using optimized camera poses? Whereas methods like MonoSDF[58], FastSURF[19] do not need fused TSDF in the test time and the results are quite comparable if not better.\n\n2. Ideally, the attention function could be also provided with the features from the grids so that the function could learn to decide upon the fusion weights also based on the features which are also being optimized. In this process, the learnt features would be considered when deciding the weights required for fusing the occupancies. This would help the network to actually reason about the fusion based on the geometric understanding learned so far.\n", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper, barring a few places, is well written.\n\n2. This paper has identified the limitation in supervising neural networks that learn implicit functions for dense 3D scene reconstruction using depth images which is difficult in supervising predictions for points whose depth values are not available as they belong to holes in the depth images.\n\n3. Authors have put in a reasonable effort to analyse the proposed method with good qualitative and quantitative analysis.\n", "weaknesses": "1. The paper lacks any substantial contribution\n\nExplanation:\n\na. It relies on hybrid representation for learning implicit functions for dense 3D reconstruction which has been very well explored before in the same context ([48], [58], [19])\n\nb. The use of pre-trained MLP decoders for predicting occupancy from trilinearly interpolated feature vectors from multi-resolution grids has been inspired by NICE-SLAM[65].\n\nc. The TSDF for the scene is obtained by fusing the posed depth images using the off-the-shelf method.\n\nd. The only contribution of this paper is towards learning an attention function that predicts the weights which are used to linearly fuse the predicted occupancies and occupancy obtained from a TSDF prior.\n\n\n2. Given the way the attention function is learned which is the primary contribution of this work, it almost looks unnecessary :\n\nExplanation: As explained in the method section and also schematically shown in the pipeline figure (fig 1), the attention function takes the sum of the predicted low and high-resolution grid occupancies and occupancy obtained by TSDF prior (normalized between 0 and 1) to predict the weights for the linear combination of the two (eq. 2, eq 3., figure 1.).\n\nThis means the attention function only chooses to learn the weights (alpha and beta, eq. 2, 3) by looking at only two floating point numbers which are between 0 and 1 based on errors in RGB and depth renderings. The RGB and Depth renderings are obtained from the final linear combination of the two occupancies (eq. 2, 3) and their quality would be inversely proportional to the rendering errors.  I don't see what else the attention function can learn to pay attention to other than which one of the inputs is larger.\n\nIn which case, how would it be any different from simply using the softmax operator to combine the two occupancies instead of learning two weights? \n\nHad the attention function taken any feature from grids to reason about how well the network can represent the scene , it could make better decision as which is more important, predicted occupancy or TSDF prior occupancy. Accordingly, it would predict weights for the fusion of the two. \n\n3. The method is affected by the very problem it is trying to solve:\nExplanation: The paper claims that missing depth values are a problem when using them raw for learning implicit functions for scene reconstruction. The fused TSDF prior would help mitigate that. However, from Figure 2 it is clear that TSDF prior could also have holes where the network has to rely on its own learned occupancy. This shows that the idea of using TSDF is itself affected by the very problem it is trying to mitigate.\n\n4. The paper also proposes that the missing depth due to occlusion leads to difficulty while supervising using depth images. However, when posed RGB-D frames of the scene are available, the points that are occluded in one frame would be visible in some other frames anyway. Consider, for example, the classic NeRF [27] paper or [Ref_DS], the novel view renderings are not affected by occlusions in different frames as the occluded ones are anyway visible in other views.\n\n5. The proposed method is very similar to [19] in the sense that both rely on fused TSDF of the scene then it would be interesting to see how the proposed method fairs with [19]. The qualitative results of [19] look quite similar to the ones shown in this paper which motivates me to also consider a quantitative comparison between the two.\n\n\n6. The paper points the reviewer to refer to supplemental material for more results. But there is no supplemental material pdf for reference (the supplemental material also has the paper in it).\n\n\n[Ref_IN] Kim, Mijeong, Seonguk Seo, and Bohyung Han. \"Infonerf: Ray entropy minimization for few-shot neural volume rendering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[Ref_DS] Deng, Kangle, et al. \"Depth-supervised nerf: Fewer views and faster training for free.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Misunderstanding in summary\n\n* “...a particular point on a sampled ray for supervision. ”\n\nWe do not use the interpolated occupancy to supervise the inferred occupancy due to the inaccuracy or incompleteness of TSDF. \n\n* “... TSDF will explain such points the best”\n\nFig.3 in our manuscript shows that, within bandwidth, TSDF may describe less (such as areas in blue) due to its inaccuracy. \n\n2. Contributions\n\nOur contribution on the learning framework and the attentive depth fusion prior should get recognized.\n\nOne is to show how we should use attention as a balance. Our ablation studies in Tab.7 and Tab.11 showed that it is great to use attention within the bandwidth, and use the inferred low resolution occupancies outside the bandwidth without attention.\n\nThe other is attention modeling. Using the interpolated and inferred occupancies without conditions performs the best in ablation studies in Tab.10 and Fig.8. Along with our framework, we believe how we should arrange focus is also worth spreading to the community to benefit the related research.\n\nWe did not claim the widely used modules are our contributions. \n\n3. The attention network\n\n3.1 Not always focus on the larger occupancy\n\nWe report analysis in Fig.1 in the rebuttal. We sample points on the GT mesh, and get the attention weights in Fig.1(a). At each point, we show its distances to the mesh from the TSDF in Fig.1(b) and the mesh from the inferred occupancies in Fig.1(c), respectively. Fig.1(d) indicates where the former is smaller than the later.\n\nThe high correlation between Fig.1(a) and Fig.1(d) indicates that the attention network focuses more on the occupancy producing smaller errors to the GT surface. \n\nInstead, the red in Fig.1(e) indicating where the interpolated occupancy is larger than the inferred occupancy is not correlated to the one in Fig.1(a). The uncorrelation indicates that the attention network does not always focus on the larger occupancy input.\n\n3.2 We used softmax\n\nWe would like to point out another reviewer sZZA’s misunderstanding. We did not directly regress two attention weights but used softmax to produce attention weights (L166-167). \n\n3.3 More attention alternatives\n\nBeyond Fig.8 in the manuscript, we additionally reported more alternatives in Fig.2 in the rebuttal.\n\n* Softmax normalization without MLP.\n* Using the larger occupancy or with maximum weight.\n* Using features as conditions. \n\n\nComparisons in Fig.2 in the rebuttal show that all these alternatives degenerate the performance. \n\n4. Holes in Fig.2 do not affect our method\n\nHoles in Fig.2 are invisible areas due to the absence of RGBD scans but not the missing depth values on depth images. We claimed that the TSDF can present missing depth values because of overlaps of neighboring views. However, we did not claim the TSDF can reveal invisible areas. \n\nMoreover, the TSDF does not supervise our implicit representations, so its incompleteness is OK to us. Geometry priors or volume rendering can infer occupancies without TSDF in the invisible area.\n\n5. Occlusion-awareness is more important in SLAM than view synthesis\n\nWe did not claim that the missing depth is caused by occlusion (L33). \n\nVolume rendering relies on multiview consistency constraints to infer occlusion in each single view. Current methods randomly sample rays on different views, randomly sample points on each ray, and expect overlaps among these points to possess multiview consistency constraints in rendering. However, the randomness makes the constraints inefficient to get. It becomes even more difficult in SLAM since one view may only be used once as a rendering target during training if it is not a key frame or far away from the current frame that is being processed, and only views before the current time step can be accessed. However, interpolation on the TSDF can directly estimate coarse occupancy for all points on a ray. It is not affected by the randomness in sampling, consistent across views, and much more occlusion-aware than single depth supervision.\n\n\n6. Difference to FastSurf [19]\n\n* [19] directly uses the TSDF as supervision.\n\n* [19] can only work in multiview reconstruction but not SLAM. \n\n* [19] can not track camera poses.\n\n* Our advantages are detailed in Fig.4 and Tab.1 in the rebuttal.\n\n7. Supplemental materials\n\nWe are sorry for submitting a wrong supplemental material pdf. We will be happy to provide any important details.\n\n8. Network difference to simply softmax\n\nAs we explained in 3.2, we indeed used softmax as a layer of MLP. Please also see our comparisons in 3.3.\n\n9. Error accumulation in TSDF\n\nFor a tradeoff between the accuracy and the efficiency, we use the most updated camera poses to fuse depth incrementally.\n\nOur solution contains a pre-fusion and an after-fusion stage. After the tracking procedure at timestep t, the pre-fusion stage first fuses the t-th depth image into a TSDF A that fused all depth images in front using the estimated t-th camera pose. This leads to a TSDF B. The mapping procedure uses the TSDF B in learning and also gets an updated t-th camera pose. Then, the after-fusion stage re-fuses the t-th depth image into the TSDF A using the updated t-th camera pose. The new TSDF A will be used at the timestep t+1. We will add a discussion on this in our revision.\n\n10. Occupancy with the maximum weight\n\nPlease see our replies in 3.3. Fig.6 in the rebuttal indicates that the inaccuracy and incompleteness make the TSDF not a good direct supervision in our method.\n\n\n11. Limitations\n\n11.1 Depth images during test\n\nLike MonoSDF, FastSurf, and NICE-SLAM, our method is also overfitting-based, all of which need RGBD images to conduct test time optimization (a stage called training but only uses a single test sample). Thus, using RGBD during test time should be a fair evaluation.\n\nLearning implicitly is to learn implicit representations for highly fidelity surface reconstruction.\n\n11.2 Feature condition for attention\n\nPlease see our replies in 3.3.\n"}]

Author rebuttal:
We thank reviewers for their valuable comments and highlighting our novel and interesting idea (REVA x87y, 3uRZ), extensive evaluations and analysis (REVR QXhf, x87y, 3uRZ, sZZA), well-motivated and sound technical method (REVR QXhf), clear presentation (REVR QXhf), and well-written manuscript (REVR x87y, 3uRZ, sZZA).

We are also sorry for uploading the wrong supplemental material. Thus, reviewers missed some important details about experimental settings, discussions on results, visualizations, and more ablation studies. We will be happy to provide any details in our following discussion period. We will upload the supplemental material if our submission could get accepted.

We respond to each review below and include a rebuttal with tables and figures here.

We will release all reviews, discussion, and our rebuttal even if our paper gets rejected.


Meta-review: This paper proposes a novel method to utilize fused depth geometry as prior for learning-based reconstruction.  The feedback during the rebuttal phase addressed the majority of the concerns raised by the reviewers. Subsequently, the evaluations from all reviewers are largely positive. In agreement with the reviewers' consensus, the AC believes this study is valuable and worth publishing.
Decision: Accept (poster)