VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Authors: Pengchong Hu
Zhizhong Han
Venue: ICML 2025 poster (2025)
Keywords: 3D Gaussian Splatting
view-tied 3D Gaussians
RGBD-SLAM
Primary area: applications->computer_vision
TL;DR: We introduce view-tied 3D Gaussians, a novel representation for RGBD SLAM that improves scalability by tying Gaussians to depth pixels, reducing storage needs and enabling finer detail representation.

Abstract: Jointly estimating camera poses and mapping scenes from RGBD images is a fundamental task in simultaneous localization and mapping (SLAM). State-of-the-art methods employ 3D Gaussians to represent a scene, and render these Gaussians through splatting 
for higher efficiency and better rendering. However, these methods cannot scale up to extremely large scenes, due to the inefficient tracking and mapping strategies that need to optimize all 3D Gaussians in the limited GPU memories throughout the training to maintain the geometry and color consistency to previous RGBD observations. To resolve this issue, we propose novel tracking and mapping strategies to work with a novel 3D representation, dubbed view-tied 3D Gaussians, for RGBD SLAM systems. View-tied 3D Gaussians is a kind of simplified Gaussians, which is tied to depth pixels, without needing to learn locations, rotations, and multi-dimensional variances. Tying Gaussians to views not only significantly saves storage but also allows us to employ many more Gaussians to represent local details in the limited GPU memory. Moreover, our strategies remove the need of maintaining all Gaussians learnable throughout the training, while improving rendering quality, and tracking accuracy. We justify the effectiveness of these designs, and report better performance over the latest methods on the widely used benchmarks in terms of rendering and tracking accuracy and scalability. Please see our project page for code and videos at https://machineperceptionlab.github.io/VTGaussian-SLAM-Project.

VTGaussianSSLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Pengehong Hu?
Zhizhong Han
Abstract
1. Introduction
SLAM is an important task in computer vision (Keetha et al., 2024; Zhu al., 2022), 3D reconstruction (Wang et. 2023; Hu & Han, 2023), and robotics (Adamkiewicz et al., 2022). SLAM methods resolve the computational problem of mapping unknown environments while tracking camera locations. Traditional SI.AM methods usually use RGB or RGBD images to sense 3D surroundings and track spatial relationships among different frames. The resulting 3D maps are usually represented by 3D point clouds, which are discrete and not friendly to downstream applications that require continuous geometry representations, such as geometry modeling, editing, and virtual reality.
Jointly estimating camera poses and mapping scenes from RGBD images is a fundamental task in simultaneous localization and mapping (SLAM). of-te-of-the-at methods employ 3D Gaussians to represent a scene, and render these Gaussians through splatting for higher efficiency and better rendering. However, these methods cannot scale up to extremely large scenes, due to the inefficient tracking and mapping strategies that need to optimize all 3D Gaussians in the limited GPU memories throughout the training to maintain the geometry and color consistency to previous RGBD observations. To resolve this issue, we propose novel tracking and mapping strategies to work with novel 3D representation, dubbed view-tied 3D Gaussians, for RGBD SLAM systems. View-tied 3D Gaussians is a kind of simplified Gaussians, which is tied to depth pixels, without needing to learn locations, rotations, and multi-ding malti-dimensional variances. Tying Gaussians to views not only significantly saves storage but also allows us to employ many more Gaussians to represent local details in the limited GPU memory. Moreover, the need of maintaining all Gaussians learnable throughout the training, while improving render- ing quality, and tracking accuracy. We justify the effectiveness of these designs, and report better performance over the latest methods on the widely used benchmarks in terms of rendering and tracking accuracy and scalability. Please see our project page for code and videos at https///machinneercppoolbal httpappperrppppllaaiiubbbb.i/i/e github.io/VTGaus: ian-SLAM-Projeet..
Due to the continuity and the ability to represent arbi- trary topology and appearance, neural representations have been employed with differentiable rendering in SI.AM sys- tems (Zhang et al., 2023; Xinyang et al., 2023; Teigen et al., 2023; Sandström et al., 2023b; Sucar et al., 2021; Wang et al., 2023). Previous methods (Zhang et al., 2023; Xinyang et al., 2023; Teigen et al., 2023; Hu & Han, 2023) learn neural radiance fields (NeRF) (Mildenhall et al., 2020), a continuous implicit function modeling scene geometry and appearance, enabling image rendering for scene mapping and camera tracking. More recently, 3DGS (Kerbl al., 2023) was proposed for high-quality and real-time rendering, which also provides novel perspective for time-sensitive SLAM problems (Matsuki et al., 2024; Huang et al., 2024b; Yan et al., 2024; Yugay et al., 2023; Sandström et al., 2024), 3DGS can efficiently render 3D Gaussians that model radi- ance fields explicitly using differentiable rasterization which is faster than the ray tracing in NeRF. However, maintaining numerous 3D Gaussians cover the whole scene while ensuring color and geometry consistency across previous frames often leads to poor rendering in tracking and map- ping. This obstacle makes 3DGS still hard to scale up to extremely large scenes in SLAMM, remaining the challenge of improving the rendering quality, tracking accuracy, and scal- ability of 3DGS tracking cameras and mapping scenes.
strategies remove
Machine Perception Lab, Wayne State University, Detroit, USA. Correspondence to: Zhizhong Han <h312h@ wdy.e.edu>.
To overcome this challenge, we propose RGBD SLAM system with splatting view-tied 3D Gaussians. Our method introduces a novel point-based volume representation, dubbed view-tied 3D Gaussians, to represent the color and
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

VTGausiaa-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
(a) Tracking at the head frame
(b) Mapping the head frame
(c) Tracking regular frame Sectian
(d) Mapping regular frame
Saafoto
Section
Sodiotio
ggl
{g)
Soctio 6.1
Sater Sime
ig
Soatin hud
Saeal There
Section
(g)
Sollia Jo!
"iitonn BAST viw
(g)
Scotoon
Leamable View-Tied Gaussians
Frozen View-Tied Gaussians
Figure 1. Overview. (a) and (c) are tracking strategies, while (b) and (d) are mapping strategies. Please refer to Sec. 3.1 for more details.
geometry of the scene, dedicated to reducing storage but pursuing better rendering quality. Different from the origi- nal 3D Gaussians (Kerbl et al., 2023), we tie 3D Gaussian to each pixel on the depth, which makes their positions only determined by depth and camera poses, without need of learning and storing their locations and also the density con- trol. Mcanwhile, we simplify an ellipsoid 3D Gaussian as sphere, without saving rotations and multi-dimensional vari- ances as well, which saves more storage for initiating even more Gaussians for describing more details. With view-tied 3D Gaussians, our novel tracking and mapping strategies can be conducted in rendering and optimizing the Gaussians that are merely related to the most recent views, rather than all Gaussians in the scene. These benefits enable us to use many more Gaussians to represent each frame, just keep the most relevant Gaussians in the limited GPU memories, and remove the need of maintaining the consistency of scene representations to keyframes. This ability scales up the size of scenes that we can handle and also significantly improves the rendering quality, even if just using simplified Gaussians. Our visual and numerical evaluations show our advantages over the latest methods. Our main contributions are listed below.
et al., 2021; Zou et al., 2022; Sun et al., 2021; Li et al., 2023) from multi-view images through volume rendering.
Besides RGB images, we can also leverage depth (Yu et al., 2022: Azinović et al., 2022; Zhu al., 2022: Zhang et al.. 2024c) and normals (Yu et al., 2022; Wang al., 2022b; Guo et al., 2022; Patel et al., 2024) as rendering supervi- sion to infer more geometry details. With the emergence of 3DGS (Kerbl et al., 2023; Moenne-Loccoz et al., 2024), we can learn implicit representations or accurate depth maps for reconstruction (Zhang et al., 2024b; Huang et al., 2024a; Yu et al., 2024; Wolf et al., 2024; Fan et al., 2024; Zhang et al., 2024a; Charatan et al., 2023; Lin & Li, 2024; Chen et al., 2024). However, these methods require accurate camera poses, which differs from SLAM methods a lot.
SLAM with Volume Rendering. With multi-view con- sistency, multi-view stereo (MVS) (Schönberger & Frahm, 2016; Schönberger et al., 2016) estimate dense depth maps from multiple RGB images. With the demand for novel view synthesis, recent SLAM methods render depth maps and estimate continuous implicit representations (Zhang et al., 2023; Xinyang et al., 2023; Teigen et al., 2023; Sandström et al., 2023b; Sucar et al., 2021; Wang et al., 2023; Sand- ström et al., 2023a;b; Hu et al., 2023). They use RGBD images as rendering supervision, and learn implicit repre- sentations through learning radiance fields. Some other methods also leverage priors like segmentation priors (Kong et al., 2023; Haghighi et al., 2023), depth fusion priors (Hu & Han, 2023), or object-level priors (Kong et al., 2023), which improve the performance in tracking and mapping.
We propose view-tied Gaussian splatting that signifi- cantly reduces storage but improves rendering quality with 3DGS in SLAM.
We introduce novel RGBD SLAM algorithm with view-tied Gaussian splatting. Our tracking and map- ping strategies remove the need of holding and optimiz- ing all Gaussians in memory throughout the training, which improves the scalability of 3DGS SLAM.
SLAM with 3D Gaussian Splatting. Because of the ren- dering efficiency and higher rendering quality, more recent methods used 3DGS to differentially render images (Keetha et al., 2024; Matsuki al., 2024; Huang et al., 2024b; Yan et al., 2024; Yugay et al., 2023; Sandström et al., 2024). Although these methods adopt different tracking and map- ping strategies, they need to maintain all Gaussians covering the scene in the limited GPU memories and optimize all Gaussians throughout the training to keep color and geom- etry consistency to all previous views. This fact limits the number of Gaussians that they can use and makes it hard to scale up extremely large scenes. Unlike these meth- ods, our method employed simplified Gaussians, without
We report the state-of-the-art results on widely used benchmarks over the latest 3DGS-based SLAM.
2. Related Work
Multi-view Reconstruction. Recent multi-view recon- struction methods aim to learn neural implicit represen- tations (Park et al., 2021; Müller et al., 2022; Rückert et al., 2021; Sara Fridovich-Kan Keil and Alex Yu et al., 2022: Zhu et al., 2022; Azinović al., 2022; Wang et al., 2022a; Bozic
2

---

VTGausiianSLLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
storing locations, rotations, or multi-view variances, saving more room for more Gaussians represent either more de- tails or larger scenes, Our view-tied Gaussians combine the advantages of Gaussian representations in current SLAM systems (Keetha et al., 2024; Yugay et al., 2023).
in this section for renderings with higher quality.
For mapping the scene using the latest frame, if the latest frame is the head of a new section in Fig. 1 (b), we initialize Gaussians by centering them at all pixels with valid depth values. If the latest frame is not a head but a regular frame in an existing section in Fig. (d), we only initialize Gaussians as complement in areas where pixels have valid depth values and the existing Gaussians in the current section cannot cover. Fig. 3 visualizes the view-tied Gaussians initialized at both the head frame and the following regular frames in a section. We learn these Gaussians to maintain their color and geometry consistency to all frames in the same section and some previous frames with overlaps.
More complicated systems (Liso et al.. 2024; Zhu et al., 2024; Bruns et al., 2024; Sandström al., 2024) also lever- age loop closure in the optimization. However, detecting loop closures among views needs pretrained priors and is sensitive to image quality, which also differs from ours.
Gaussian Alignment. Aligning 3D Gaussians to some entities was also employed in some other works (Yinghao et al., 2024; Zhang et al., 2024a; Gao et al., 2024; Luiten et al., 2024; Seidenschwarz et al., 2024; Zakharov et al., 2024). Gaussians in these methods are either not related to camera positions or with many attributes. Instead, our view- tied Gaussians relate Gaussians' positions with the camera poses and use simplified attributes, dedicated to improving the efficiency and scalability of SLAM systems.
3.2. View-tied Gaussians
Our view-tied Gaussians aim to achieve memory efficiency in SLAM, which enables us to improve the rendering quality by using many more Gaussians to represent local details, Specifically, we simplify an ellipsoid Gaussian g into a sphere, which only includes a color e E RII3 radius as the variance E IR, and an opacity E R and removes the 4-dimensional rotation, the 3-dimensional location, the other 2-dimensional variances from the original 3D Gaus- sians, saving 64.3% (9/14) storage in total.
3. Methods
3.1. Overview
Fig. 1 illustrates our tracking and mapping strategies. We organize view-tied 3D Gaussians from several consecutive frames as a section so that we can keep as many Gaussians as the GPU memory allows to represent a local area, access these Gaussians more efficiently, and more importantly, en- able more robust completion of missing depth by utilizing neighboring frames' depth. In each section, we mark the first frame as a head to differentiate it from regular frames for different Gaussian initialization strategies in mapping.
We remove the need of learning and storing locations by ty- ing a Gaussian g at each pixel with valid depth value on the depth map. We center g at the 3D location back-projected from the depth value. Thus, the positions of Gaussians are only determined by the depth and the corresponding camera pose, which can be adjusted in the camera tracking proce- dure. For the i-th frame with a RGB V; and a depth map D;, we will initialize Gaussians {g]} on D:-
With view-tied Gaussians, we manage to keep learnable Gaussians that are the most relevant to the latest frame in the GPU memory. This ability removes the need of main- taining a list of keyframes and optimizing all Gaussians rep- resenting the scene to maintain their spatial and appearance consistency to all keyframes, which widely employed in the latest SLAM systems (Wang et al., 2023; Hu & Han, 2023; Keetha et al., 2024; Zhu et al., 2022), leading to the key of scaling up to much larger scenes.
Since there may be missing depth values on some depth maps, organize Gaussians from every N neighboring frames into section St, so that we can use Gaussians from neighboring frames to cover the missing area on one specific depth map in the same section. Based on the concept of sec- tions, we only optimize Gaussians in one section containing the latest view, and frozen Gaussians other sections. This not only enables to use more Gaussians to represent local details, but also removes the need to maintain the appear- ance and geometry consistency to all the previous frames or keyframes. This is a key to improving our rendering quality, leading to better performance in tracking and mapping, and scaling up to larger scenes.
For tracking the latest frame, we select Gaussians in a sec- tion, render them from the camera pose initialized by the constant speed assumption, and optimize the pose by min- imizing rendering errors to the latest frame. If the latest frame is the head of new section in Fig. 1 (a), we select the Gaussians in a certain section in front according to the visibility. These selected Gaussians maintain the spatial consistency in a long view sequence and reduce the error cu- mulation. the latest frame is not a head but a regular frame in the current section in Fig. (c), we select the Gaussians
3.3. Tracking Cameras
We will estimate the camera pose P. of the latest frame [V, D.} first. When tracking cameras, we keep all Gaus- sians in the scene fixed, and merely optimize the pose p; by minimizing rendering errors with respect to (V,, D.).
3

---

VTGAUSiia-SLLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Head Frame
Section
Scction 73
Section 74
Figure 2. Illustration of selecting overlapping section. We show Gaussian centers and colors each section.
illustrate this issue in Fig. 4, where we use Gaussians in the current section and render pretty good images for tracking. At each frame out of a 2000 frame video, the average error of relative pose to the previous frame is pretty small, while the average error relative to the whole trajectory is getting larger and larger. To resolve this issue, we maintain the spatial consistency of the current section by rendering Gaussians {g} in certain previous section So for tracking the head frame while rendering Gaussians {g}* in the same section S₁. for tracking regular frames.
The Head Frame
Rogular Frame
Regular Frame
Rogular Frame
Figure Initialization view-tied Gaussians a section.
If the latest frame {V,, } is a head starting a new section Sg. we select one section S in front of Sk. and render Gaussians in So into RGB V and depth D using the pose P,. We optimize Pa to minimize rendering errors,
Selecting Overlapping Section S In each section, we select a section S for the head frame in terms of overlap and visibility. We first set up an overlap candidate view list with an interval of N₃ frames. For head frame {V,, D,), we project the depth D; from the initialized pose to each one frame in the overlap candidate view list. We count the number of visible points to each candidate view, and select the candidate views that have more visible points than a threshold yy. We eventually select the most front sections containing least one of the selected candidate views. We use the most front criterion relieve the impact caused by the error cumulation.
min V. 1/ll + sW. - P.
(1)
where (V,, D() splat((1g) (1g) S" are rendered im- ages by splatting {g) in section So using the camera pose Ps. W; is a mask which removes pixels either without depth values or uncovered by the rendering of {g} or invisible to sections So, and a and B are balance parameters.
Otherwise, if the latest frame (V, D is regular frame in the current section Sg, we will optimize the camera pose p: using the same equation above but rendering the Gaussians {g)* in the current section S₁. into (V. D.).
Then, we adopt pre-tracking strategy to finalize the se- lection. We use each one of the sections to conduct the tracking for several iterations, respectively. We eventually select section that produces the minimum rendering er- ror as S Here, we do not splat Gaussians from all the 3 sections together for rendering since multiple sections seldom learned together in the mapping procedure, which may degenerate rendering quality. Fig. 2 illustrates the 3 section candidates selected for a head frame, where all the 3 sections are highly related to the head frame. We finally render Gaussians in section 73 to track the head frame.
This design aims to find a balance between the rendering quality and the spatial consistency of the current sec- tion to previous frames in a long image sequence. Obviously, ren- dering Gaussians in the same section will produce better renderings since neighboring frames usually have larger overlaps with the latest frame. Although better renderings are helpful for more accurate camera pose estimations, the higher accuracy is merely meaningful relative to the neighboring frames, resulting in a significant cumulation of pose errors relative to the whole camera trajectory. We
Relative Pose AVG.
Absolute Poso Envr(AVV)
Maiiiine Plnc Eroor AVC
Absulutc Puse EirocAva)
130
Frame
Figure 4. Issue of pose error cumulation.
Visibility Check. We determine the visibility of a 3D point to a specific view if its projected depth is within 1% of the interpolated depth at its projection on the depth map. For visibility to a section S with W, in Eq. 1, we consider the head frame, the frame in the middle, and the last frame in So to calculate three visibility masks. We use the union of these three visibility masks as the visibility mask to a

---

VTGaussian-LLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
section. Please see Fig. 10 for more details.
3.4. Mapping Scenes
0.77
We initialize Gaussians and learn attributes of Gaussians in the mapping procedure. In each section, we first initialize Gaussians at all pixels on the depth map the head frame, and then complement Gaussians at pixels uncovered by the current Gaussians' renderings on regular frames. To maintain the appearance and geometry consistency to frames in both the same section and nearby overlapping sections, we adopt different strategies to learn Gaussian attributes.
RGi Image
Mapping Loss
0.15
814
Una ney
1uth naar
Mar
60
If the latest frame (V,, D is a head frame starting new the Gaussians {g)* initialized the
Number of Mapping ltrrations
section Sk.
Figure 5. Illustration of optimizing view-tics tied Gaussians initialized head frame. Error maps are shown at different iterations.
latest frame, the Gaussians {g)*-1 in the previous sec- tion S and the Gaussians {g} in the section S hav- ing the largest overlaps with the latest frame to render VV/,D3] = splat([(g)** }*, (9) "l,)). We mini- mize the rendering errors with respect to observations,
Implementation Details. For neighboring views in a sec- tion Sg, we choose N = 40 on Replica (Straub et al., 2019). N = 30 on TUM-RGBD (Sturm et al., 2012), N 30 or N = 50 on ScanNet (Dai al., 2017a), and N = 100 on ScanNet++ (Yeshwanth et al., 2023), according to image resolutions and mapping iterations. During tracking, we use every frames as a candidate view for overlapping section selection, i.e. N1 = 5. Specifically, when mapping a regu- lar frame in an existing section Sg. we will choose 0.5 as mask threshold determine if current Gaussians in S₃ cover pixel, if they do not, we will initialize a Gaussian as complement. To work with depth and RGB images with different quality and resolutions, we set p 0.8, 7 = 0.2, and = 1.0 to balance the mapping loss terms while we set {a, 3} = {0.5, 0.025) on Replica, {0.5, 1.0} on TUM- RGBD and ScanNet++, {0.5, 0.9} on ScanNet to balance the tracking loss terms. More details of hyperparameters are provided in the supplementary materials.
min VIVV, (Ø)*
V.. +o)+UI D; Dill, (2)
where Lg is the SSIM loss, U, is a mask which removes pix- els without valid depth values, p. T, a are balance weights. Only {g} th are learnable, and all other Gaussians are fixed. This rendering can maintain the appearance and geometry consistency to the nearby overlapping sections.
Otherwise, if the latest frame (V,, D is regular frame in the current section Sg. we first splat the Gaussians {g} in the current section into silhouette images to find the un- covered area where we initialize Gaussians. Then, we splat *g)* render from a random view Pj of {V, D that is in front of (V,, Di} in this section, i.e., (V7, Dj} = (g)* Pý), where (Vj, D2} E Sy and j S i. We still use Eq. to optimize {g]²; where we select a random view as a rendering target to maintain the appearance and geome- try consistency to frames in the same section. We visualize the rendered images during mapping head frame in Fig. 5, where the rendering error is progressively minimized.
splot(1g))
Dataset and Metrics. We conduct evaluations on several widely used benchmarks, including Replica (Straub et al., 2019), TUM-RGBI (Sturm et al., 2012), ScanNet (Dai et al., 2017a), and ScanNet++ (Yeshwanth et al., 2023). Here Replica synthetic dataset with high-fiellity 3D reconstruction of indoor scenes. We evaluate on the widely used RGBD sequence from eight scenes captured by Su- car (2021) with accurate trajectories. TUM-RGBD, Scan- Net, and ScanNet++ are real-world datasets. Note that Scan- Net++ is not a dataset designed for SLAM tasks, some sudden large motions are occurring in the DSLRRcaptured sequences, we follow previous methods (Yugay et al., 2023; Zhu et al., 2024) and only employ the first 250 frames of each scene in evaluations, which present smooth trajectories.
3.5. Bundle Adjustment
We also conduct bundle adjustment to jointly estimate the camera pose and learn Gaussians when mapping the head frame in each section. We still use the Eq. 2 in the opti- mization, but also back-propagate gradients to update the camera pose of the head frame. We do not use bundle ad- justment on each regular view in section for stabilizing the optimization in both tracking and mapping.
4. Experiments and Analysis
We evaluate the accuracy of estimated cameras at each frame and the rendering quality from either the observed or unobserved view angles. For tracking accuracy, we use the root mean square absolute trajectory error (ATE RMSE) (Sturm et al., 2012) as a metric, Regarding render-
We only report average results in this section, evaluations on each scene can be found in the supplementary materials. Please also watch our video for more renderings.
5

---

VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
NICE-SLAM
SplaTAM
Gaussian-SLAM -LLAM
Ours
GT
off4
Rm0
Figure 6. Visual comparisons in reconstruction on Replica.
Table 1. Tracking comparisons in ATE RMSE [cm] on Replica. * denotes use of pre- Irainrd data-driven priors.
Noural Implicit Fields
3D Gaussian Splatting
Methods
NICE-SLAM
DF-Prior
Vox-Fusion
ESLAM
Poiet-SLAM
Loopy-SLAM-
SSpaTAM
GS-SLAM
GaussianSLLA
LoopSplat-
CG-SLAM
Ours
Avg.
1.95
1.81
0.65
0.63
0.52
0.29
0.36
0.50
0.31
0.26
0.27
0.28
NICE-SLAM
SplaTAM
Gaussian-SL SLAM
Ours
GT
Table 2. Rendering results in PSNR t. SSIM f. and LPIPS I on three Datasets. denotes use pre-trained data-driven priors.
Datawet
Replice
TUM
SoowNir
1
PSNR
Nourai Implicir Feids
Figure 7. Error map comparisons in rendering on Replica.
NICE-SLAM
24.42
0.809
0.233
14.86
0.614
0.441
17.54
0.621
0.548
Von-Fusion
24.41
0.501
0.236
16.45
0.677
0.673
0.504
ESLAM
0.245
15.26
0.658
0488
ing quality, we measure PSNR, SSIM (Wang et al., 2004), and LPIPS (Zhang et al., 2018). Similar to (Sandström et al., 2023a; Liso et al., 2024; Zhu et al., 2024; Yugay et al., 2023), all the rendering metrics are computed by rendering the full resolution images along the estimated trajectory every frames. Additionally, we also obtain the meshes of scenes by marching cubes (Lorensen Cline, 1987) following similar procedure in (Sandström al., 2023a). Then measure the reconstruction performance with Fi-score, the harmonic mean of the Precision (P) and Recall (R), using distance threshold of em for all evaluations. We also use the depth L. metric to measure the rendered mesh depth error at sampled novel views as in (Zhu et al., 2022).
Poirt-LAM
35.17
0,9775
0.124
16,62
0.696
0.526
19,82
0.751
0.514
Loopy SLAM-
35.47
0.981
0.109
12.94
0.489
0.645
15.23
0.629
0.671
3D Gaurzian Splaming
SyliTAM
34.11
0.970
0.100
22.80
0.891
0.178
19.14
0.716
0.358
0.923
LoopSplat-
36.61
0.985
0.112
22.72
0.873
0,259
24.92
0.845
0.25
0.996
0.012
0.961
0.108
methods with pose graph optimizations Loo (Liso et al., 2024) and LoopSplat (Zhu et al., 2024), and CG- SLAM (Hu al., 2024) usually reported higher tracking aceuracy, which however to most SLAM methods without using priors.
a fair experimental setting
4.1. Comparisons
Baselines. We compare our method with the latest RGBD SLAM methods, including NeRF-based RGBD SLAM methods: NICE-SLAM (Zhu et al., 2022), Vox- Fusion (Yang et al., 2022), ESLAM (Johari et al., 2023),
Results Replica. We first report results scenes in Replica. We compare with the state-of-the-art NeRF. based and GS-based SLAM methods in camera tracking in Tab. 1, mapping scenes with rendered images in Tab. 2, and reconstruction in Tab. 3.
DF-Prior (Hu & Han, 2023), Co-SLAM (Wang et al., 2023), and Point-SLAM (Sandström et al., 2023a), and 3DGS- based RGBD SLAM methods: SplaTAM (Keetha et al., 2024), MonoGS (Matsuki al., 2024), GS-SLAM (Yan al., 2024), and Gaussian-SLAN (Yugay al., 2023). Note that Point-SLAM (Sandström et al., 2023a) requires ground truth depth images as an input to guide sampling when rendering, which is an unfair advantage compared to other methods. Moreover, relying on data-driven priors, such as pre- pre-trained NetVLAD models (Arandjelović et al., 2016), loop closure detection and visibility check, SLAM
Tab. shows that our method can estimate camera poses more accurately than sfate-of-the-at NeRF-based methods, such as NICE-SLAM (Zhu et al., 2022), DF-Prior (Hu & Han, 2023), and Point-SLAM (Sandström al., 2023a), due to higher quality renderings produced by view-tied Gaussians. Our view-tied Gaussians also show advan- tages over the original Gaussians used in SplaTAM (Keetha et al., 2024), Coussian-SLAN (Yugay et al., 2023), and GS SLAM (Yan et al., 2024) in terms of using much more
6

---

VTGaussian-ILAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Table Reconstruction results in D-L1 [em] J and F1 [%]] t on Replica.
denotes use of pre-trained data-driven priors.
Neural Implicit Fields
3D Gaussian Splatting
NICE-SLAM
Vox-Fusion
ESLAM
Co-SLAM
Point SLAM
Loupy-SLAM+
SplaTAM
GS-SLAM
Gaussian-SLAM
LoupSplate
Ours
D-LIU FIf
2.97
2.46
1.18
2.59
0.44 89.8
0.35
0.72
1.16
0.68
0.51
0.53
43,9
52.2
79.1
69.7
90.8
86.1
70,2
88,9
90.4
90.0
Table Tracking comparisons in ATE RMSE + [cm] on TUM-RGBD. * denotes use of pre-trained data-driven priors.
Neural Implicit Fields
3D Gaussian Splatting
Methods
NICE-SLAM
Vox-Fusion
Point-SLAM
Loopy-SLAM-
SplaTAM
GS-SLAM
Baussian-SLAM,
LoopSplat
CG-SLAM+
Ours
Avg.
13.3
10.3
3.0
2.9
3.3
3.7
2.9
2.3
2.0
2.6
NICE-SLAM
SplaTAM Gaussian-SLay
Ours
GT
tions are not in high resolution and with good quality, our method still produces the best tracking accuracy. Regard- ing the rendering, our view-tied Gaussians show even more advantages over the other methods in Tab. Visual com- parisons in Fig. 8 highlight our improvement over the other methods. Compared to previous GS-based SLAM methods, our method can use many more Gaussians tied at each pixel on depth images to fit sudden color change without needing to maintain the consistency of Gaussians over the whole scene to a set of keyframes throughout the training, which enables Gaussians focus more on local details.
Results ScanNet. Our evaluations in camera tracking and mapping scenes with rendering views are reported in Tab. 5 and Tab. 2, respectively. We produce the most accu- rate tracking performance in terms of average performance. Based on the camera poses, our method also significantly improves the rendering quality on ScanNet, as shown in Fig. 9. The rendering improvement also justifies our advan- tages of using view-tied Gaussians on real-captured scenes, Besides good ability of recovering appearance details, with motion blur and low image quality in real images, our view- tied Gaussians can also limit these negative impact just on several neighboring views but not on all Gaussians in the scene during mapping.
Figure 8. Visual comparisons in rendering on TUM-RGBD.
Gaussians to represent local details in a more efficient man- ner, leading to better renderings to compare with the obser- vations during tracking. However, relying on data-driven priors, LoopSplat (Zhu et al., 2024) reported more accurate camera tracking in terms of average accuracy, while our method does not need any priors.
Due to the ability of using more Gaussians describe local details, our method produces the best rendered images, as shown in Tab. 2. Error map comparisons in Fig. 7 highlight our rendering quality, which is much better than others, especially in areas with sudden color changes. Because of better renderings, our methods also produce the most accurate reconstruction in Tab. 3. We follow the previous method (Sandström et al., 2023a) to render depth maps from the estimated camera poses, and fuse these depth maps into TSDF for reconstruction. Visual comparisons in Fig. 6 show that we can recover more accurate geometry details.
NICE-SIAM
SplaTAM
Gaussian-ST.N
Qurs
Results on TUM-RGBD, We report our results on the TUM- RGBD dataset in camera tracking in Tab. 4 and rendering in Tab. 2. We follow previous methods (Zhu et al., 2022: Keetha et al., 2024; Yugay et al., 2023; Liso et al., 2024;
Zhu et al., 2024; Sandström et al., 2023a) and evaluate our method on the 3 widely used scenes in TUM-RGBD.
Comparisons in camera tracking in Tab. 4 show that GS- based methods estimate camera poses more accurately than Nese-based methods. Although the input RGBD observa-
Figure 9, Visual comparisons in rendering on ScanNet.
7

---

VTGaussian-SLMM RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Visi-Head
(b) Visi-Middle
(c) Visi-Last
Visibility
(e) Projection
( Visi. Prroj.
(g)
Rendered RGBD with Masks
Figure 10. Visualization of visibility and Gaussian projections on a head frame during tracking in a section.
(a) Overlapping Sections for Head Frames
(b) Overlapping Sections for Regular Frames
Gaussian Centers: Head Fraiess
Regular Frames
Figure 11. Visualization of Gaussian centers with colors in the selected overlapping section for tracking (a) head frames and (b) regular frames. The Gaussian centers nearby are shown without color.
Table 5. Tracking comparisons ATE RMSE L [cmu on ScanNet. denotes use pre-trained data-driven priors.
Neural Implicit Fields
3D Gaussian Splatting
Methods
NICE-SLAM
Vox-Fusion
Point-SLAM
Loopy-SLAM-
SplaTAM
Gaussian-SLANe
LoopSplat
CG-SLAM-
Ours
Avg.
10.7
26.9
12.2
7.7
11.9
15.4
7.7
8.1
11.3
Table 6. Tracking comparisons in ATE RMSE, [em] on ScanNet++ * denotes use of pre-trained data-driven priors.
Neural Implicit Fields
3D Gaussian Splatting
Methods
Point-SLAM
ESLAM
Loopy-SLAM=
SplaTAM
Gaussian-Si SLAM
LoopSplat*
Ours
Avg.
511.24
22.14
113.63
89.41
2.68
2.05
1.55
Results on ScanNet++. We report tracking results on the widely used scenes in ScanNet++ in Tab. 6. Compared to GS-based methods, our methods can estimate more accu- rate camera poses thanks the more accurate renderings. Regarding novel view synthesis, please refer to our supple- mentary materials for more details.
fied Gaussians ('iso w/o VT"). Without tying Gaussians to depth maps, we need store Gaussian locations, which limits the number of Gaussians we can use, degenerating the rendering quality. The comparisons show that our view- tied Gaussians not only significantly reduce the size of each Gaussian (number of parameters) but also achieve good rendering quality with our tracking and mapping strategies.
4.2. A Ablation Studies and Analysis
Section Length. The number of frames section also a factor impacting the performance. Tab. shows that too few or too many frames in one section will degenerate the perfor- mance if we do not adjust other parameters like optimization iterations during tracking mapping. Too few frames will increase the possibility of cumulating camera pose errors while changing into the next section. Instead, too many frames will need more iterations during mapping to learn Gaussians well. We cannot use large number of Gaussians
We justify the effectiveness of each design synthetic and real scenes in Replica (Straub et al., 2019) and TUM- RGBD (Sturm et al., 2012).
We conduct experiments to highlight the effect of view-tied Gaussians in Tab. We replace our Gaussians with ellipsoid Gaussians with either fixed ("aniso + w/ VT") learnalle locations ("aniso w/o VT"). We also show the effect of learnable locations with our simpli-
Gaussians.
8

---

VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Table 7. Ablation study attributes of 3D Gaussians (aniso: anisotropic Gaussians, iso: isotropic Gaussians, VT: view-tied Gaussians).
Metric
aniso + w/o VT
aniso+ w/ VT
iso + w/o VT
iso + w/ VT (Ours)
PSNRT [dB) ATE RMSE + [em]
37.62
38.46 25.65
36.73
39.95
19.60
21.45
0.22
Param. /Gaussian
14
11
8
5
Table Ablation study on the length of section S, overlap selecting strategy, and visible mask.
Length of section S
Overlap scction selecting strategy
Visibility mask
Metric
20
40 (Ours)
60
80
100
nearest
largest
multiple
Ours
w/o
w/ (Ours)
PSNR (dB)
39.92
39.95
39.34
38.87
38.56
38.52
38.74
38.03
39.95
30.40
30.51
ATE RMSE! + [cm]
0.25
0.22
0.23
0.26
0.24
0.30
0.34
0.28
0.22
6.8
4.4
Table 9. Runtime and Memory Usage on Replica.
Method
NICE-SLAN SLAM
Saitt-SLAM
SplaTAM
Gaussian-
Ours
Tracking/Frame(s)
1.06
1.11
2.70
0.83
1.92
Mapping/Frame(s)
1.15
3.52
4.89
0.93
2.43
Total Num of Gaussians Max Num of Gaussians
5832K
32592K
97823
5832K
1983
2664K
if setting the length of the section 1, which degenerates the rendering and tracking performance. We visualize some overlapping sections during tracking in Fig. 11.
time for tracking and mapping one frame in Tab. 9 on single NVIDIA RTX4090. Our runtime is comparable to other GS-based methods. Regarding the number of Gaus- sians, we show a great advantage over other methods. We can initiate many more Gaussians over the whole scene than other GS-based methods for better rendering. Meanwhile, around each frame, we still have good control of the num- ber of Gaussians that limited GPU memory, which finds good balance between scalability and limited hardware resources. This enables us to handle much larger scenes in more frames with more Gaussians than other SLAM methods.
Overlapping Section Selection Strategy. Our strategy of selecting overlapping sections also important for good renderings in tracking the head frame. Our selection based on visibility and preference the most front sections signifi- cantly reduces the impact of pose error cumulation. We com- pare this strategy with selecting the nearest section ("near- est"), the section with the largest overlaps ("largest"), or multiple sections ("multiple") that add the nearest section to the selection we selected. The comparisons Tab. 8 show that our selection strategy achieves the best performance.
maximize the usage of the
5. Conclusion
Visibility. We consider visibility in both tracking and map- ping to reduce the error brought by unseen occluded areas. Using no visibility in the loss function will degenerate the performance, as shown in Tab. 8, since the error in the un- seen area will not be minimized by adjusting camera poses or optimizing Gaussian attributes. We visualize the visibility masks and projection masks in Fig. 10. The visibility masks of head frame to the first, the middle, and the last frame in the selected overlapping section are shown in Fig. 10 (a)-(c), respectively. The overall visibility to the overlapping section is the union of these 3 visibility masks in Fig. 10 (d). The silhouette produced by projections of Gaussians in the same section is shown in Fig. 10 (c). We use the intersection of visibility mask and silhouette mask in Fig. 10 (f) to weight rendered RGBD images in Fig. 10 (g).
We propose VTGaussian-SLe SLAM to improve the perfor- mance of SLAM with 3D Gaussian splatting in terms of rendering quality, tracking accuracy, and scalability. We showed that our view-tied Gaussians can significantly save storage so that we can maintain large amount of these Gaussians in the limited GPU memory for either higher rendering quality or larger areas. Our tracking and map- ping strategies take good advantage of these benefits, which allows to merely maintain and optimize Gaussians that contribute to the most recent views a lot. This ensures the high rendering quality for the latest views, and leads to more accurate camera tracking and mapping. We justified the effectiveness of each design and reported visual and numerical evaluations illustrate our advantages over the latest SLAM methods.
Runtime, Storage, and Scalability. We report the average
9

---

VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Acknowledgements
Fan, L.,, Yang, Y., Li, M., Li, H., and Zhang, Z. Trim 3d gaussian splatting for accurate geometry representation. arXiv preprint arXiv:2406.0744
This project was partially supported by an NVIDIA aca- demic award and a Richard Barber research award.
2024.
Gao, L.., Yang, J., Zhang. B.-T., Sun, J.-M., Yuan, Y.-.,,
Impact Statement
Fu, H., and Lai, Y.-K. Mesh-based gaussian splatting for real-time large-scale deformation, 2024. URL https: //arxiv. org/ abs /2002. 04796.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must specifically highlighted here.
Geiger, A., Lenz, P., and Urtasun, Are we ready for autonomous driving? the kitti vision benchmark suite. In Computer Vision and Paltern Recognition, 2012.
References
Guo, H., Peng, S., Lin, H., Wang, Q., Zhang, G., Bao, H., and Zhou, X. Neural 3d scene reconstruction with the manhattan wooldd assumption. IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Adamkiewicz, M., Chen, T., Caccavale, A., Gardner, R., Culbertson, P., Bohg. J., and Schwager, M. Vision- only robot navigation in neural radiance world. IEEE Robotics and Automation Letters, 7(2)4606-4613, 2022.
Haghighi, Y., Kumar, S., Thiran, J.-P., and Gool, L. V. Neu- ral implicit dense semantic slam, 2023.
Arandjelović, R., Gronat, P.. Torii, A., Pajdla, T., and Sivic, NetVLAD: CNN architecture for weakly supervised place recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Hu, J., Mao, M., Bao, H., Zhang, G., and Cui, Z. Cp-slam: Collaborative neural point- slam system. In Neural Information Processing Systems (NeurIPS), 2023.
Azinović, D., Martin-Brualle R., Goldman, D. B., Nießner, M., and Thies, J. Neural rgb-d surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, pp.
Hu, J., Chen, X., Feng, B., Li, G., Yang, I.., Bao, H., Zhang, G., and Cui, Z. Cg-slam: Efficient dense rgb-d slam in a consistent uncertainty-aware 3d gaussian field. arXiv preprint arXiv:2403.1609) 2024.
2022.
Bozic, A., Palafox, P., Thies, J., Dai, A., and Niessner, M. Transformerfusiin: Monocular rgb scene reconstruction using transformers. Advances in Neural Information Pro- cessing Systems, 2021.
Hu, P. and Han, Z. Learning neural implicit through volume rendering with attentive depth fusion priors. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Bruns, L., Zhang, J., and Jensfelt, P. Neural graph mapping for dense slam with efficient loop closure. arXiv preprint arXiv:2405.0363: 036333, 2024.
Huang, B., Yu, Z., Chen, A., Geiger, A., and Gao, S. 2d gaussian splatting for geometrically accurate radi- ance fields. In SIGGRAPH 2024 Conference Papers. Association for Computing Machinery, 2024a. doi:
Charatan, D., Li, S., Tagliasacchi, A., and Sitzmann, V. pix- elsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In arXiv, 2023.
10.1
Huang, H., Li, L., Hui, C., and Yeung, S.-K. Photo-slam:
Real-time simultancous localization and photorealistic mapping for monocular, stereo, and rgb-d cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Chen, D., Li, H., Ye, W., Wang, Y., Xie, W., Zhai, S., Wang, N., Liu, H., Bao, H., and Zhang, G. Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction, 2024. URL https:://///o //tp//aavvv//////////e//eaaaav.a org/ abs/2406 06521.
Johari, M. M., Carta, C., and Fleuret, F. ESLAM: Effi- cient dense slam system based on hybrid representation of signed distance fields. In Proceedings of the IEEE international conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T. A., and Nießner, M. Scannet: Richly-annotated reconstructions of indoor scenes. CoRR, abs/1702.04405, 2017a.
Keetha, N., Karhade, J., Jatavallabhula, K. M., Yang, G., Scherer, S., Ramanan, D., and Luiten, J. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, 2024.
Dai, A., Nießner, M., Zollöfer, M., Izadi, S., and Theobalt, C. Bundlefusion: Real-time globally consistent 3d recon- struction using on on-the-fh the-fly surface re-integration. ACM Transactions on Graphics, 2017b.
10

---

VTGaussian-SLMM, RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Kerbl, B., Kopanas, G., Leimkühler, T., and Drettakis, G. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL https::/ inria.fr/ fungraph/ -a-gaddsianss wawsssiannsspltttiee
Park, K., Sinha, U., Barron, J. T., Bouaziz, S., Goldman, D. B., Seitz, S. M., and Martin-Brualla, R. Nerfies: De- formable neural radiance fields. IEEE International Con- ference on Computer Vision, 2021.
Patel, A., Laga, H., and Sharma, O. Normal- guided detail- preserving neural implicit functions for high-fidelity 3d surface reconstruction, 2024. http:: aarxiv..
Kong, X., Liu, S., Taher, M., and Davison, A. J. vmap: Vectorised object mapping for neural field slam. arXiv preprint ar arXiv:2302.0/88. 2023.
org/ abs / 2406 04861.
Li, M., Liu, S., Zhou, H., Zhu, G., Cheng, N., Deng, T., and Wang, H. Sgs-slam: Semantic gaussian splatting for neural dense slam, 2024. URL https::////// /arxiv.. org/ org/abs// abs/ 2402. 03246.
Rückert, D., Franke, L., and Stamminger, M. Adop: Approximate differentiable one-pixel point rendering. arXiv:2110.06633, 2021.
Sandström, E., Tateno, K., Oechsle, M., Niemeyer, M., Van Gool, L.., Oswald, M. R., and Tombari, Splat- slam: Globally optimized rgb-only slam with 3d gaus- sians. arXiv preprint arXiv:2405.16544 2024.
Li, Z., Lyu, X., Ding, Y., Wang, M., Liao, Y., and Liu, Y. Rico: Regularizing the unobservable for indoor composi- tional reconstruction, 2023.
Lin, A. and J. J. Direct learning of mesh and appearance via 3d gaussian splatting, 2024. URL https: //arxiaravvv/// org/ org/abs/ haabbbb//2055 06945.
Sandström, E., Li, Y., Van Gool, L., and R. Oswald, M. Point-slam: Dense neural point cloud-based slam. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2023a.
Liso, L., Sandström, E., Yugay, V., Van Gool, L., and Os- wald, M. R. Loopy-slam: Dense neural slam with loop closures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 20363- 20373, 2024.
Sandström, E., Ta, K., Gool, L. V., and Oswald, M. R. Uncle-slam: Uncertainty learning for dense neural slam, 2023b.
Sara Fridovich-Keti and Alex Yu, Tancik, M., Chen, Q.. Recht, B., and Kanazawa, A. Plenoxels: Radiance fields without neural networks. IEEE Conference on Com- puter Vision and Pattern Recognition, 2022.
Lorensen, W. E. and Cline, H. E. Marching cubes: A high resolution 3D surface construction algorithm. Computer Graphics, 21(4):163-16) 1987.
Luiten, J., Kopanas, G., Leibe, B., and Ramanan, D. Dy- namic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.
Schönberger, J. L. and Frahm, J.-M. Structure- froeee--fommmae motion revisited. In JEEE Conference on Computer Vision and Pattern Recognition, 2016.
Matsuki, H., Murai, R., Kelly, P. H. J., and Davison, A. Gaussian Splatting SLAM. 2024.
Schönberger, J. L., Zheng, E.,, Pollefeys, M., and Frahm, J.- M. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision, 2016.
Mildenhall, B., Srinivasan, P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
Seidenschwarz, J., Zhou, Q. Duisterhof, B., Ramanan, D., and Leal-Taixé, L. Dynomo: Online point tracking by dynamic online monocular gaussian reconstruction. 2024. URL https: / /atpiviivvvv////.///aa////aaavaa wwwp////g/aa.a././///// 02104.
Moenne-Loccoo N., Mirzaei, A., Perel, O., de Lutio, R., Esturo, J. M., State, G., Fidler, S., Sharp, N., and Gojcic, Z. 3d gaussian ray tracing: Fast tracing of particle scenes. ACM Transactions on Graphics and SIGGRAPH Asia, 2024.
Straub, J., Whelan, T., Ma, L., Chen, Y.. Wijmans, E., Green, S., Engel, J. J., Mur-Artal, R., Ren, C., Verma, S., Clark- son, A., Yan, M., Budge, B., Yan, Y., Pan, X., Yon, J., Zou, Y., Leon, K., Carter, N., Briales, J., Gillingham, T., Mueggler, E., Pesqueira, I.., Savva, M., Batra, D., Stras- dat, H. M., Nardi, R. D., Goesele, M., Lovegrove, S., and Newcombe, R. A. The replica dataset: digital replica of indoor spaces. CoRR, abs/1906.05799 2019.
Müller, T., Evans, A., Schied, C., and Keller, A. Instant neural graphics primitives with a multiresolution hash encoding, arXiv:2201.08) 05989, 2022.
Park, J., Zhou, Q.-Y., and Koltun, V. Colored point cloud registration revisited. In 2017 IEEE International Con- ference on Computer Vision (ICCV), pp. 143-152, 2017. doi: 10.1 109/CCCV2011.25.
Sturm, J., Engelhard, N., Endres, F., Burgard, W., and Cre- mers, D. benchmark for the evaluation of rgb-d slam

---

VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp- 573-580, 2012. doi: 10.1109/OOO, 201....5..212..887773
15M1AR55827.2222.00666 URL http: //bp:://dx.// doi. org/10. 110011.11100//1m1555222222 2022. 00066.
Yeshwanth, C., Liu, Y.C.., Nießner, M., and Dai, A. Scan- net++: A high-fide high-fidelity dataset of 3d indoor scenes. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Sucar, E.,, Liu, S., Ortiz, J., and Davison, J. J. imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 6229-6238, 2021.
Yinghao, X., Zifan, S., Wang, Y., Hansheng, C., Ceyuan, Y., Sida, P., Yujun, S., and Gordon, W. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation, 2024.
Sun, J., Xie, Y., Chen, L., Zhou, X., and Bao, H. Neu- ralRecon: Real-time coherent 3D reconstruction from monocular video. IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Yu, Z., Peng. S., Niemeyer, M. Sattler, T., and Geiger, A. MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction. ArXiv, abs/2022.00665, 2022.
Teigen, A. L., Park, Y., Stahl, A., and Mester, R. Rgb-d mapping and tracking in a plenoxel radiance field, 2023.
Wang, H., Wang, J., and Agapito, L. Co-slam: Joint coordi- nate and sparse parametric encodings for neural real-time slam, 2023.
Yu, Z., Sattler, T., and Geiger, A. Gaussian opacity fields: Efficient adaptive surface reconstruction in unbounded scenes. ACM Transactions on Graphics, 2024.
Wang. J., Bleja, T., and Agapito, L.. Go-surf: Neural feature grid optimization for fast, high-fidelity rgb-d surface re- construction. In International Conference on 3D Vision, 2022a.
Yugay, V., Li, Y., Gevers, T., and Oswald, M. Gaussian- slam: Photo-realistic dense slam with gaussian splatting, 2023.
Wang, J., Wang, P., Long, X., Theobalt, C., Komura, T., Liu, L., and Wang, W. NeuRIS: Neural reconstruction of in- door scenes using normal priors. European Conference on Computer Vision, 2022b.
Zakharov, E., Sklyarova, V., Black, M. J., Nam, G., Thies, J., and Hilliges, O. Human hair reconstruction with strand- aligned 3d gaussians. ArXiv, Sep 2024.
Zhang, K., Bi, S., Tan, H., Xiangli, Y., Zhao, N., Sunkavalli, K., and Xu, Z. Gs-Irm: Large reconstruction model for 3d gaussian splatting. European Conference on Computer Vision, 2024a.
Wang, Z., Bovik, A., Sheikh, H., and Simoncelli, E. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing. 13 (4):600-612 2004. doi: 10.1 00.1109/T1P2200.888983
Wei. J. and Leutenegger, S. Gsfusion: Online rgb-d mapping where gaussian splatting meets tsdf fusion, 2024. URL attpss//aaee org/ abs/ 2408 12677.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as perceptual metric, 2018. URL hotttttss//axiii abs/ 1 1801 03924.
Wolf, Y., Bracha, A., and Kimmel, R. Gs2mesh: Surface reconstruction from gaussian splatting via novel stereo views. arXiv preprint wtxXi:2404.0181000 2024.
Zhang, W., Liu, Y--.., and Han, 7. Neural signed distance function inference through splatting 3d gaussians pulled on zero-level set. In Advances in Neural Information Processing Systems, 2024b.
Xinyang, L., Yijin, L., Yanbin, T., Hujun, B., Guofeng, Z., Yinda, 7.,, and Zhaopeng, C. Multi-modal neural radiance field for monocular dense slam with light- weight tof sensor. In International Conference on Computer Vision (ICCV), 2023.
Zhang, W. Shi, K., Liu, Y-S.,, and Han, Z. Learning un- signed distance functions from multi-view images with volume rendering priors. European Conference Com- puter Vision, 2024c.
Yan, C., Qu, D., Xu, D., Zhao, B., Wang, Z., Wang. D., and Li, X. Gs-slam: Dense visual slam with 3d gaussian splatting. In CVPR, 2024.
Zhang, Y., Tosi, F., Mattoccia, S., and Poggi, M. Go-slam: Global optimization for consistent 3d instant reconstruc- tion. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), October 2023.
Yang, X., Li, H., Zhai, H., Ming, Y. Liu, Y., and Zhang, G. Vox-fusion: Dense tracking and mapping with voxel-based neural implicit representation. In 2022 IEEE International Symposium on Mixed and Aug- mented Reality (ISMAR), Dec 2022. doi: 10. 109/
Zhu, L., Li, Y., Sandström, E., Huang, S., Schindler, K., and Armeni, I. Loopsplat: Loop closure by registering 3d gaussian splats, 2024.
12

---

VTGaussianSILAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Zhu, Z., Peng, S., Larsson, V., Xu, W., Bao, H., Cui, Z Oswald, M. R., and Pollefeys, M. Nice-slam: Neural implicit scalable encoding for slam. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Zou, Z., Huang, S., Cao, Y., Mu, T., Shan, Y., and Fu, H. Mononeuralfusion: Online monocular neural 3d recon- struction with geometric priors. CoRR, abs/2209.151555 2022.
13

---

VTGaussianSIIAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Supplementary Material
et al., 2024; Zhu al., 2024; Wei & Leutenegger, 2024; Li et al., 2024) and conduct experiments three scenes of TUM-RGBD, six scenes of ScanNet, and five scenes of ScanNet++ ((a) b20a2 (b) 8b5caf3398, (c) fb0 5e 13d11, (d) 2e7481 2d00, (e) 281be1 7764) to evaluate our performance.
In this supplementary material, we will cover more details about the implementation and results on each scene, Addi- tionally, we also show more results regarding Novel View Synthesis on ScanNet++ (Yeshwanth et al., 2023).
A. Implementation Details
We report numerical comparisons in camera tracking in each scene in Replica (Straub et al., 2019) in Tab. 10, in TUM- RGBD (Sturm et al., 2012) in Tab. 11, in ScanNet (Dai et al., 2017a) in Tab. 13, and in ScanNet++ (Yeshwanth et al., 2023) in Tab. 15. The comparisons show that our methods can estimate more accurate camera poses in most scenes.
We implemented VTGaussian- SLAM in Python using the PyTTorch framework, and ran all experiments on NVIDIA RTX4090 GPUs. During mapping, we will initialize view- tied Gaussians from input RGBD images. Following (Keetha et al., 2024), we initialized the radius of each Gaus- sian r using the following equation:
Moreover, we report reconstruction comparisons in each scene in Replica (Straub al., 2019) in Tab. 19, We uti- lize depth LI and F1-score metrics to evaluate the mesh obtained by marching cubes (Lorensen & Cline, 1987) fol- lowing a similar procedure in (Sandström et al., 2023a). Compared to the latest methods, our methods can recover more accurate geometry, although Point-SLAM (Sandström et al., 2023a) requires ground truth depth images as input to guide sampling when rendering, which is an unfair ad- vantage compared to other methods. Specifically, we also show visual comparisons regarding camera tracking and reconstruction in Fig. 12. Please refer to our supplementary video for more details of this comparison.
DGT
Here Dar is the ground-truth depth, and f is the focal length. As for the learning rate of 3D Gaussians, we set Treolor 0.0025 for the color, Breadius = 0.005 for the radius, and Propncity = 0.05 for the opacity separately. For camera tracking, we initialize the pose using the constant speed assumption
Replica (Straub et al., 2019), TUM- RGBD (Sturm et al., 2012), and ScanNet (Dai et al.. 20)7 Specifically, on ScanNet++ (Yeshwanth et al., 2023), since it is not dataset originally designed for SLAM tasks, some sudden large motion changes are occurring in the DSLR- captured sequences, we follow previous methods (Zhu et al., 2024; Yugay et al., 2023) to utilizing multi-scale RGBD odometry (Park et al., 2017) to help the pose initialization if the rendering error with the pose initialized by constant speed assumption is 50 times larger than the average of the rendering loss for previous frames after the tracking optimization. We set the learning rate of pose to Ir = 0.0004 and Irtrane
We also report comparisons in rendering in each scene from the training views in Replica (Straub et al., 2019), TUM- RGBD (Sturm et al., 2012), ScanNet (Dai et al., 2017a), and ScanNet++ (Yeshwanth et al., 2023) separately. Due to our view-tied strategy, there will be more 3D Gaussians used for rendering, which leads to better rendering quality compared to the latest methods as shown in Tab. 18, Tab. 12, Tab. 14, and Tab. 16.
0.002 on Replica, Iroot = 0.002 and Irtrans 0.002 on TUM-RGBD and ScanNet, and Irrot- = 0.001 and Ir trane 0.01 on ScanNet++ separately. Additionally, we set the overlapping threshold y = 0.26 on TUM-RGBD, and = 0.24 on ScanNet and ScanNet++ separately.
Large-scale scenes Results. We additionally report our performance on extremely large scenes, such as city-level scenes in KITTI (Geiger et al., 2012). Since many moving objects exist in KITTI sequences, we only select part of the sequences to evaluate our tracking and rendering perfor- mance in Tab. 20 and Tab. 21. The comparisons show that our methods can estimate more accurate camera poses with high quality rendering performance. Meanwhile, we report memory consumption on KITTI in Tab. 21, Each method uses the most Gaussians until no improvement can be made. We use little bit more memory, but we manage to use more Gaussians to produce much better rendering.
B. More Results
Per-scene Results. We present more detailed results on each scene in Replica (Straub et al., 2019), TUM- RGBD (Sturm et al., 2012), ScanNet (Dai et al., 2017a), and ScanNet++ (Yeshwanth et al.. 2023). Replica is a syn- thetic dataset, whereas TUM-RGBD, ScanNet, and Scan- Net++ are real-world datasets. TUM-RGBI were captured using an external motion capture system, while ScanNet uses poses from BundleFusion (Dai et al., 2017b), and Scan- Net++ utilizes a laser scan to register the images for ac- quiring corresponding camera poses. We follow previous methods (Keetha et al., 2024; Yugay et al., 2023; Matsuki
C. Novel View Synthesis
We evaluate the performance in novel view synthesis on ScanNet++ (Yeshwanth et al., 2023). The testing views in ScanNet++ are from held-out views, which are much dif-
14

---

VTGaussian-ILAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
ferent from the training views. To evaluate PSNR on all post-processing step after training is
sian ("Aniso.") and learnable Gaussian locations ("w/o VT"". The comparison shows that our Gaussians can produce the minimum rendering errors.
test views, finished. In post-processing. we use all training views to re- fine all trained sections {Sx) Specifically, for each training view, we find all its overlapping sections and refine all these sections using this view. Compared to the latest methods (Zhu et al., 2024; Yugay et al., 2023), which take 10K itera- tions to refine their global map. our refinement only needs 1K iterations for novel view synthesis. In testing, given novel view, we will find all overlapping sections Sg. and concatenate all these sections to render the novel view im- age. Tab. 17 shows that our methods can obtain comparable novel view synthesis results to the latest methods. We also show qualitative rendering results of the training views in Fig. 13 and novel view synthesis Fig. 14.
Many more advantages can be shown with our estimated camera poses during mapping in Fig. 19. Our method can recover more details the scene and produce more accurate renderings.
E. Code
Please see our project page for code at http:/acheeepppeepioiiiiiiiibbid SLAMM-roject,.
F. Video
D. More analysis and visualization
We present more visualization in our video, such visual comparisons and visualization of the optimization. Please watch our video for more details.
D.I. NOISE DEPTH
Our view-tied Gaussians can also resist the impact brought by noise in depth. Although Gaussians are fixed depth with noise, Gaussian splatting is flexible enough overfit the current frame and neighboring frames by tuning other attributes like color, opacity, and shape. Our results show that depth noises do not significantly impact the rendering performance. Meanwhile, we try to optimize the position of Gaussians along the ray direction, but we do not find an obvious improvement in rendering performance. We report additional results in Tab. 22. We also report visual com- parison using either fixed Gaussians movable Gaussians (along the ray) in Fig. 15.
D.2. ISSUE OF POSE ERROR CUMULATION
Here we present the effectiveness of our tracking strategy. As shown by average pose accuracy (ATE RMSB) in Fig. (a) and (b), using overlapping selection for rendering in tracking can prevent the estimated camera pose from drifting away from the trajectory caused by the error cumulation. The absolute pose error at each frame is shown in Fig. 16 (c) and (d). Since our method can render pretty good images, we produce small pose errors relative to the previous view in Fig. 16 (e) and (f).
D.3. VISUALIZATION OF 3D GAUSSIANS
Fig. 18 visualizes optimized 3D Gaussians. Our methods initialize much denser 3D Gaussians to represent the whole scene due to the view-tied strategy. With more 3D Gaussians, our methods can render more realistic images.
We also report a comparison different kinds of Gaussians in Fig. 17. We employ the same number of Gaussians, but highlight the performance of our Gaussians in rendering with different specifications, such using ellipsoid Gaus-
15

---

VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
i
NICE-SLAM
SplaTAM
Gaussian-SLAM
Ours
Figure 12. Visual comparisons in camera tracking on Replica. also show error maps on reconstructions. Please refer to our video for more complete comparison during scanning.
Table 10. Tracking performance comparisons in ATE RMSE [cm] on Replica (Straub et al., 2019). pre-trained data-driven priors.
indicates methods relying on
Method
Rm0
Rm1
Rm2
OfEO
Off1
Off2
OfE3
Of
Avg.
Neural Implicit Fields
NICE-SLAM (Zhu et al., 2022)
1.69
2.04
1.55
0.99
0.90
1.39
3.97
3.08
1.95
DF-Prior (Hu & Han, 2023)
1.39
1.55
2.60
1.09
1.23
1.61
3.61
1.42
1.81
Vox-Fusion (Yang al., 2022)
0.27
1.33
0.47
0.70
1.11
0.46
0.26
0.58
0.65
ESLAM (Johari et al., 2023)
0.71
0.70
0.52
0.57
0.55
0.58
0.72
0.63
0.63
Point-SL.M (Sandström et al., 2023a)
0.61
0.41
0.37
0.38
0.48
0.54
0.72
0.63
0.52
Loopy-SLAM* (Liso et al., 2024)
0.24
0.24
0.28
0.26
0.40
0.29
0.22
0.35
0.29
3D Gaussian Splatting
SplaTAM (Keetha et al., 2024)
0.31
0.40
0.29
0.47
0.27
0.29
0.32
0.55
0.36
GS-SLAM (Yan et al., 2024)
0.48
0.53
0.33
0.52
0.41
0.59
0.46
0.70
0.50
Gaussian- SLAM (Yugay et al., 2023)
0.29
0.29
0.22
0.37
0.23
0.41
0.30
0.35
0.31
LoopSplat> (Zhu et al., 2024)
0.28
0.22
0.17
0.22
0.16
0.49
0..0
0.30
0.26
CG-SLAME (Hu et al., 2024)
0.29
0.27
0.25
0.33
0.14
0.28
0.31
0.29
0.27
Ours
0.22
0.26
0.19
0.28
0.26
0.34
0.25
0.43
0.28
16

---

VTGaussian-ILAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Table 11. Tracking performance comparisons in ATE RMSE + [cm] on TUM-RGBD (Sturm et al., 2012). * indicates methods relying on
pre-trained data-driven priors.
Method
fr1/desk fr2/xyz
fr3/office
Avg.
Neural Implicit Fields
NICE-SLAM (Zhu et al., 2022)
4.3
31.7
3.9
13.3
Vox-Fusion (Yang et al., 2022)
3.5
1.5
26.0
10.3
Soint-LLAM (Sandström et al., 2023a)
4.3
1.3
3.5
3.0
Loopy- AAMM (Liso et al., 2024)
3.8
1.6
3.4
2.9
3D Gaussian Splatting
SplaTAM (Keetha et al., 2024)
3.4
1.2
5.2
3.3
GS-SLAM (Yan et al., 2024)
3.3
1,3
6.6
3.7
Gaussian-SLAN (Yugay et al., 2023)
2.6
1.3
4.6
2.9
LoopSplat* (Zhu et al., 2024)
2.1
1.6
3.2
2.3
CG-SLAM* (Hu et al., 2024)
2.4
1.2
2.5
2.0
Ours
2.4
1.1
4.4
2.6
Table Rendering performance comparison PSNR 1. SSIM t. and LPIPS on TUM-RGBD (Sturm et al., 2012). indicates methods relying pre-trained data-driven priors.
Method
Metric
fr1/desk
fr2/xyz #r3/office
Avg.
Neural Implicit Fields
PSNR
13.83
17.87
12.89
14.86
NICE-SLAM (Zhu et al., 2022)
SSIM
0.569
0.718
0.554
0.614
LPIPS]
0.482
0.344
0.498
0.441
PSNR
15.79
16.32
17.27
16.46
Vox-Fusion (Yang et al., 2022)
SSIM
0.647
0.706
0.677
0.677
LPIPS.
0.523
0.433
0.456
0.471
PSNR 11.29
17.46
17.02
15.26
ESLAM (Johari al., 2023)
SSIM
0.666
0.310
0.457
0.478
LPIPS
0.358
0.698
0.652
0.569
PSNR
13.87
17.56
18.43
16.62
Point-SLAM (Sandström al., 2023a)
SSIM
0.627
0.708
0.754
0.696
LPIPS
0.544
0.585
0.448
0.526
PSNR
12.94
Loopy-SLAMe (Liso et al., 2024)
SSIM
0,489
LPIPS
0.645
3D Gaussian Splatting
PSNR
22.00
24.50
21.90
22.80
SplaTAM (Keetha et al., 2024)
SSIM
0.857
0.947
0.876
0.893
LPIPS
0.232
0.100
0.202
0.178
PSNR
24.01
25.02
26.13
25.05
Gaussian-SLAM (Yugay et al., 2023)
SSIM
0.924
0.924
0.939
0.929
LPIPS.
0.178
0.186
0.141
0.168
PSNR
22.03
22.68
23.47
22.72
LoopSplats (Zhu et al., 2024)
SSIM
0.849
0.892
0.879
0.873
LPIPS
0.307
0.217
0.253
0.259
PSNR
27.09
33.01
30.50
30.20
Ours
SSIM
0.959
0.982
0.974
0.972
LPIPS
0.085
0.038
0.063
0.062
17

---

VTGaussianSSLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Table 13. Tracking performance comparisons in ATE RMSE + [em] on ScanNet (Dai et al., 2017a). indicates methods relying on
pre-trained data-driven priors.
Method
0000
0059
0106
01
0181
0207
Avg.
Neural Implicit Fields
LICE-SLAM (Zhu et al., 2022)
12.0
14.0
7.9
10.9
13.4
6.2
10.7
Vox-Fusion (Yang al., 2022)
68.8
24.2
27.3
23.3
9.4
26.9
Point- SLAM (Sandström et al., 2023a)
10.2
7.8
8.7
22.2
14.8
9.5
12.2
Loopy-SLAM- SLAM (Liso et al., 2024)
4.2
7.5
8.3
7.5
10.6
7.9
7.7
3D Gaussian Splatting
SplaTAM (Keetha al., 2024)
12.8
10.1
17.7
12.1
11.1
7.5
11.9
Gaussian-ST ALAM (Yugay et al., 2023)
24.8
8.6
11.3
14.6
18.7
14.4
15.4
LoopSplat= (Zhu et al., 2024)
6.2
7.1
7.4
10.6
8.5
6.6
7.7
CG-SLAMs (Hu et al., 2024)
7.1
7.5
8.9
8.2
11.6
5.3
8.1
Ours
17.8
8.7
11.8
10.5
10.6
8.6
11.3
Table 14. Rendering performance comparison in PSNR SSIM f. and LPIPS relying on pre-trained data-driven priors.
ScanNet (Dai al., 2017a).
indicates methods
Method
Metric
0COC
0059
01066
01.69
01
0207
Avg.
Neural Implicit Fields
PSNRT
18.71
16.55
17.29
18.75
15.56
18.38
17.54
NICE-SLAM (Zhu al., 2022)
SSIMt
0.641
0.605
0.646
0.629
0.646
LPIPS
0.561
0.534
0.510
0.534
0.602
0,552
0.548
PSNRT
19.06
16.38
18.46
18.69
16.75
19.66
18.17
Vox-Fusion (Yang al., 2022)
SSIM
0.662
0.615
0.753
0.650
0.666
0.696
0.673
LPIPS
0.515
0.528
0.439
0.513
0.532
0.500
0.504
PSNRT 0.687
14.48
15.44
14.56
17.32
15.29
ESLAM (Johari et al., 2023)
SSIM
0.632
0.628
0.656
0.696
0.653
0.658
LPIPS
0.449
0.450
0.529
0.486
0.482
0.534
0.488
PSNRT
19.48
16.80
18.53
22.27
20.56
Point-SLAM (Sandström et al., 2023a)
SSIM 0.806
0.765
0.676
0.686
0.823
0.750
0.751
LPIPS
0.485
0.499
0.544
0.542
0.471
0.544
0.514
PSNRT
T5.23
Loopy-SLAMe SLAM* (Liso et al., 2024)
SSIM
0.629
LPIPS
0.671
3D Gaussian Splatting
PSNRT
19.33
19.27
17.73
21.97
16.76
19.8
19.14
SplaTAM (Keetha al., 2024)
SSIM
0.660
0.792
0.690
0.776
0.683
0.696
0.716
LPIPS
0.438
0.289
0.376
0.281
0.420
0.341
0.358
PSNRT
28.54
26.21
26.26
28.60
27.79
28.63
27.70
Gaussian-SLAN (Yugay et al., 2023)
SSIM
0.926
0.934
0.926
0.917
0.922
0.914
0,923
LPIPS
0.271
0.211
0.217
0.226
0.277
0.288
0.248
PSNRf
24.99
"3.22
23.33
266.0
24.82
266.33"
24.92
LoopSplat (Zhu et al., 2024)
SSIM
0.840
0.831
0.846
0.877
0.824
0.854
0.845
LPIPS
0.450
0.400
0.409
0.346
0.514
0.430
0.425
PSNRT
31.51
30.60
31.27
32.02
29.60
31.58
31.10
Ours
SSIMI
0.957
0.974
0.975
0.962
0.946
0.961
LPIPS
0.131
0.080
0.074
0.091
0.145
0.124
0.108
18

---

VTGaussianSILAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Table 15. Tracking performance comparisons in ATE RMSE + [em] ScanNet++ (Yeshwanth al., 2023). indicates methods relying on pre-trained data-driven priors
Method
Avg.
Neural Implicit Fields
Point-SLAM (Sandström al., 2023a)
246.16
632.99
830.79
271.42
574.86
511.24
ESLAM (Johari et al., 2023)
25.15
2.15
27.02
20.89
35.47
22.14
Loopy-SLAM (Liso et al., 2024)
25.16
234.25
81.48
113.63
3D Gaussian Splatting
SplaTAM (Keetha et al., 2024)
1.50
0.57
0.31
443.10
1.58
89.41
Gaussian-SLAA SLAM (Yugay et al., 2023)
1.37
5.97
2.70
2.35
1.02
2.68
LoopSplate (Zhu al., 2024)
1.14
3.16
3.16
1.68
0.91
2.05
Ours
2.79
1.50
0.96
1.18
1.31
1.55
Table 16. Rendering performance comparison in PSNR 1 on ScanNct++ (Yeshwanth et al., 2023). * indicates methods relying on
pre-trained data-driven priors.
Method
a
d
Avg.
3D Gaussian Splatting
SplaTAM (Keetha al., 2024)
28.02
27.93
29.48
19.65
28.48
26.71
Gaussian-SLAn SLAM (Yugay et al., 2023)
30.06
30.02
31.15
28.75
31.94
30.38
LoopSplat= (Zhu et al., 2024)
30.15
30.08
30.04
28.94
31.78
30.20
Ours
32.84
31.02
32.44
31.43
33.38
32.22
Table 17. Novel View Synsthesis performance comparison in PSNR on ScanNet++ (Yeshwanth al., 2023). relying on pre-trained data-driven priors.
indicates methods
Method
b
d
Avg.
3D Gaussian Splatting
SplaTAM (Keetha et al., 2024)
23.95
22.66
13.95
8.47
20.06
17.82
GauS (Yugay et al., 2023)
26.66
24.42
15.01
18.35
21.91
21.27
LoopSplat* (Zhu et al., 2024)
25.60
23.65
15.87
18.86
22.51
21.30
Ours
25.55
24.25
16.94
18.59
21.95
21.46
19

---

VTGaussian-ILAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
SplaTAM
Gaussian-SLan SLAM
LoopSplat
Ours
b20a26 fddt
Zoom In
fb05e1 1ad1
Zoom In
Figure 13. Visual comparisons in training view rendering on ScamNet++ (Yeshwanth et al., 2023).
SplaTAM
Gaussian-SLAve SLAM
LoopSplat
Ours
b20a26 far
Zoom In
281be 281be17764 1764
Zoom In
Figure 14, Visual comparisons in novel view rendering on ScanNet++ (Yeshwanth et al., 2023).
20

---

VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Table 18. Rendering performance comparisons in PSNR 1. SSIM and LPIPS
Replica (Straub al., 2019).
indicates methods
relying on pre- trained data-driven priors.
Method
Metric
Rm0
Rm1
Rm2
Offo
Off1
Off2
Off3
Off4
Avg.
Neural Implicit Fields
PSNRT
22.12
22.47
24.52
29.07
30.34
19.66
22.23
24.94
24.42
DICE-SLAM (Zhu et al., 2022)
SSIM
0.689
0.757
0.814
0.874
0.886
0.797
0.801
0.856
0.809
LPIPS
0.330
0.271
0.208
0.229
0.181
0.235
0.209
0.198
0.233
PSNR 22.39
22.36
23.92
27.79
29.83
20.33
23.47
25.21
24.41
Vox-Fusion (Yang et al., 2022)
SSIM
0.683
0.751
0.798
0.857
0.876
0.794
0.803
0.847
0.801
LPIPS
0.303
0.269
0.234
0.241
0.184
0.243
0.213
0.199
0.236
PSNR1 25.25
27.39
28.09
30.33
27.04
27.99
29.27
29.15
28.06
ESLAM (Johari et al., 2023)
SSIM
0,874
0.89
0.935
0.934
0.910
0.942
0.953
0.948
0.923
LPIPS
0.315
0,296
0.245
0,213
0.254
0,238
0.186
0.210
0.245
PSNR1 32.40
34.08
35.50
38.26
39.16
33.99
33.48
33.49
35.17
Point-STAM (Sandström al., 2023a)
SSIM
0.974
0.977
0.982
0.983
0.986
0.960
0.960
0.979
0.975
LPIPS.
0.113
0.116
0.111
0.100
0.118
0.156
0.132
0.142
0.124
PSNR
35.47
Loopy-SLAM* (Liso et al., 2024)
SSIM
0.981
LPIPS
0.109
3D Gaussian Splatting
PSNR
32.86
33.89
35.25
38.26
39.17
31.97
29.70
31.81
34.11
SplaTAM (Keetha al., 2024)
SSIM
0.98
0.97
0.98
0.98
0.98
0.97
0.95
0.95
0.97
LPIPS
0.07
0.10
0.08
0.09
0.09
0.10
0.12
0.15
0.10
PSNR1
32.50
34.25
35.10
38.54
39.20
32,90
32.05
32.75
34.66
SGS-SLAM (Li al., 2024)
SSIM
0.976
0.978
0.981
0.984
0.980
0.967
0.966
0.949
0.973
LPIPS
0.070
0.094
0.070
0.086
0.087
0.101
0.115
0.148
0.096
PSNRT 31.56
32.86
32.59
38.70
41.17
32.36
32.03
32.92
34.27
GS-SLAM (Yan et al.. 2024)
SSIM
0.968
0.973
0.971
0.986
0.993
0.978
0.970
0.968
0.975
LPIPS
0.094
0.075
0.093
0.050
0.033
0.094
0.110
0.112
0.082
PSNR1 34.83
36.43
37.49
39.95
42.09
36.24
36.70
36.07
37.50
MonoGS (Matsuki et al., 2024)
SSIM
0.954
0.959
0.965
0.971
0.977
0.964
0.963
0.957
0.960
LPIPS
0.068
0.076
0.075
0.072
0.055
0.078
0.065
0.099
0.070
PSNRT 38.88
41.80
42.44
46.40
45.29
40.10
39.06
42.65
42.08
Gaussian-SLAM (Yugay et al., 2023)
SSIM 0.993
0.996
0.996
0.998
0.997
0.997
0.997
0.997
0.996
LPIPS] 0.017
0.018
0.019
0.015
0.016
0.020
0.020
0.020
0.018
PSNR1 33.07
35.32
36.16
40.82
40.21
34.67
35.67
37.10
36,63
LoopSplat* (Zhu et al., 2024)
SSIM 0.973
0.978
0.985
0.992
0.990
0.985
0.990
0.989
0.985
LPIPS 0.116
0.122
0.111
0.085
0.123
0.140
0.096
0.106
0.112
PSNRT 33.27
34.60
CG-SLAM= (Hu et al., 2024)
SSIM
LPIPS
PSNR1 39.95
43.06
43.13
46.88
47.20
42,14
40.99
43.35
43.34
Ours
SSIM
0.992
0.996
0.996
0.998
0.997
0.996
0.996
0.996
0.996
LPIPS
0.014
0.013
0.014
0.009
0.009
0.012
0.013
0.015
0.012
21

---

VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Ours
Movable Gaussians
Movable Gaussians
GT RGB
RGB Errormap
RGB Errormap
Visualization in 2D
GT Depth
Depth Errormap
Depth Errormap
Visualization in 3D
Figure 15. Visual comparisons on fixed Gaussians movable Gaussians (along the ray) in rendering Replica. We provide rendering results, including RGB error maps and depth error maps rendered by using either our view-tied Gaussians or movable Gaussians along the ray. Additionally, movement forward along the ray. Furthermore, we provide 3D visualization the optimized movable Gaussians (gray: central points of Gaussians fixed depth, red: central points of Gaussians optimized along the ray).
provide 2D visualization of the optimized movable Gaussians (blue: movement backward along the ray. red:
Table 19. Reconstruction performance comparison in Depth LI [em]] and F1 [%] Replica (Straub al., 2019). relying on pre-trained data-driven priors.
indicates methods
Method
Metric
Rm0
Rm2
O Fo
Off1
Off2
Off3
Cff4
Avg.
Neural Implicit Fields
Depth L1 [em]]]
1.81
1.44
2.04
1.39
1.76
8.33
4,99
2.01
2.97
NICE-SIAM (7hu al., 2022)
F1 [%] f
45.0
44.8
43.6
50.0
51.9
39.2
39.9
36.5
43.9
Depth LI [em]]
1.09
1.90
2.21
2.32
3.40
4.19
2.96
1.61
2.46
Vox-Fusion (Yang et al., 2022)
[%] f
69.9
34.4
59.7
46.5
40.8
51.0
64.6
50.7
52.2
Depth LI [em]]
0.97
1.07
1.28
0.86
1.71
1.43
1.18
ESLAM (Johari et al., 2023)
F1 [%] f
81.0
82.2
83.9
78.4
75.5
77.1
75.5
79.1
79.1
Depth L1 [em]]]
0.99
0.82
2.28
1.24
1.61
7.70
4.65
1.43
2.59
Co-SLAM
F1 [%] f
77.7
74.2
69.3
75.2
75.2
54.3
56.8
75.3
69.7
Depth 1.1 [em]]]
0.53
0.22
0.46
0.30
0.57
0.49
0.51
0.46
0.44
Point-SLAM (Sandström et al., 2023a)
[%] f
86.9
92.3
90.8
93.8
91.6
89.0
88.2
89.8
Depth LI [em]]
0.30
0.20
0.42
0.23
0.46
0.60
0.37
0.24
0.35
Loopy-SLAMs (Liso et al., 2024)
[%] f
91.6
92.4
90.6
93.9
91.6
88.5
89.0
88.7
90.8
3D Gaussian Splatting
Depth LI [em]]
0.43
0.38
0.54
0.44
0.66
1.05
1.60
0.68
0.72
SplaTAM (Keetha al., 2024)
F1 [%] f
89.3
88.2
88.0
91.7
90.0
85.1
77.1
80.1
86.1
Depth L1 [cm]]
1.31
0.82
1.26
0.81
0.96
1.41
1.53
1.08
1.16
GS-SLAM (Yan et al., 2024)
[%] f
62.9
79.9
66.8
80.0
81.6
66.0
59.2
65.0
70.2
Depth L1 [em]]
0.61
0.25
0.54
0.50
0.52
0.98
1.63
0.42
0.68
Gaussian-SLAM (Yugay et al., 2023)
[%] t
88.8
91.4
90.5
91.7
90.1
87.3
84.2
87.4
88.9
Depth LI [em]]"
00.9
0.23
0.52
0.32
0.5l
0.63
1.09
0.40
0.5T
LoopSplat (Zhu et al., 2024)
[%] f
90.6
91.9
91.1
93.3
90.4
88.9
88.7
88.3
90.4
Depth LI [em]]
0.48
0.28
0.61
0.41
0.48
0.62
0.86
0.53
0.53
Ours
[%]
90.7
91.7
90.7
93.0
90.8
88.3
87.5
87.0
90.0
22

---

VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Table 20. Tracking performance comparisons ATE RMSE
[in] on KITTI (Geiger et al., 2012).
Sequence
Gaussian-SLAM
SplaTAM
LoopSplat
Ours
00
3.02
58.83
2.22
2.06
01
77.51
84.45
74.47
29.01
05
128.88
80.39
117.43
7.74
10.60
43.82
11.39
4.54
Table 21, Rendering performance comparisons PSNR
KITTI (Geiger al., 2012).
Sequence
Gaussian- -LAM
SplaTAM
LoopSplat
Ours
00
15.51
9.82
15.82
28.54
01
15.95
12.89
14.69
30.33
05
16.22
26.48
15.98
28.19
10
15.58
25.58
14.58
27.59
Peak GPU Use (GiB)
2.74
22.37
3.56
4.79
Table 22, Impact of depth noise and movability Gaussians on the rendering performance in PSNR 1. SSIM 1. and LPIPS J on Replica (Straub al., 2019).
Metric
10% pixels w/ noises
20% pixels w/ noises
30% pixels w/ noises
Gaussians movable along ray
Ours(w/o additional noises fix)
PSNR
43.41
43.40
43.29
42.89
43,06
SSIM?
0.996
0.996
0.996
0.995
0.996
LPIPSI
0.015
0.015
0.015
0.020
0.013
23

---

VTGausiia-SSLM:: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians
Replica: office4
TUM-RGBD: fr3/office
Replica: rooml
Replica: office0
Pusiia Fos TT
Maila Fore Sead Foetii
ScanNet: seene0000
SeanNet: seene0169
(d)
Figure 18. Visualization of optimized 3D Gaussians.
(C)
(f)
Depthh
Figure 16. Comparison w/ w/o overlap selection when track-
ing.
Anisn. wio VT
Tso. + wio VT
Ton. wi VT (Ours)
SplaTAM
Gansilan-S
Rendere I0Gi8
Dift Rendered
Dirt RoGB
Diff: Depth Rendered Depth
Rendered Depsli
DiT. Depil
Figure 19, Visual comparisons of rendered images and depths. We also show maps (large rendering are shown in red). Please refer to our video for more visual comparisons rendered images.
Figure 17. Comparisons of different kinds of Gaussians (with the same number). Our view-tied Gaussians in the 3rd column can recover accurate RGB color and depth in renderings, while the ellipsoid Gaussians in the 1st column and sphere Gaussians in 2nd column produce worse rendering quality with adjustable Gaussians. Please refer to our video for complete comparison of optimization.
24


Reviews:
[{"review_id": "MEsKmCGOCi", "reviewer": "Reviewer_4tKp", "summary": "This work presents VTGaussian-SLAM, a novel method for RGB-D SLAM by a novel view-tied 3D Gaussian representation, with corresponding tracking and mapping methods. \nThis method does reduce parameter optimization (e.g., exact localization, rotation and covariance parameters), so the system can store many more Gaussians in the GPU memory for more detail and/or larger scale mapping. \nThe method is organized in terms of sections of frames. In a section, added frames spawn new Gaussians (for novel geometry coverage) or update existing ones for uniform appearance and geometry. Crucially, only the current section's (and a limited number of overlapping sections for consistency in pose) Gaussians are learnable at any point, removing the need for global consistency with all keyframes. \nResults with Replica, TUM-RGBD, ScanNet, and ScanNet++ show improvements in both camera tracking accuracy (ATE) and rendering quality (PSNR, SSIM, LPIPS), and in reconstruction metrics (depth L1, F1) over prior state-of-the-art SLAM methods based on both implicit fields (NeRF variants) and 3D Gaussian splatting.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and positive comments on our idea and evaluations.\n\n### **1. Benchmark selection and large-scale scenes**\n\nWe follow previous methods like SplaTAM to report our evaluations on the widely used benchmarks such as ScanNet++. We also show our advantages in storage complexity, which can learn 20 times more Gaussians to recover more details on all frames than the latest methods, like SplaTAM, and also overcome the obstacle of out of memory at any frame during optimization. These advantages significantly improve our capability and performance in large-scale scenes.\n\nAs requested, we additionally report our performance on extremely large scenes, such as city-level scenes in KITTI.  Due to the time limit, we only report our results on several scenes to evaluate our tracking and mapping performance below. We also report a visual comparison of rendering performance at the link: https://imgur.com/a/FOIgcrz.\n|Methods|Gaussian-SLAM|SplaTAM|LoopSplat|Ours|\n|-|:-:|:-:|:-:|:-:|\n|00 (ATE RMSE$\\downarrow$[m])|3.02|58.83|2.22|**2.06**|\n|01 (ATE RMSE$\\downarrow$[m])|77.51|84.45|74.47|**29.01**|\n|05 (ATE RMSE$\\downarrow$[m])|128.88|80.39|117.43|**7.74**|\n|10 (ATE RMSE$\\downarrow$[m])|10.60|43.82|11.39|**4.54**|\n\n|Methods|Gaussian-SLAM|SplaTAM|LoopSplat|Ours|\n|-|:-:|:-:|:-:|:-:|\n|00 (PSNR$\\uparrow$)|15.51|9.82|15.82|**28.54**|\n|01 (PSNR$\\uparrow$)|15.95|12.89|14.69|**30.33**|\n|05 (PSNR$\\uparrow$)|16.22|26.48|15.98|**28.19**|\n|10 (PSNR$\\uparrow$)|15.58|25.58|14.58|**27.59**|\n|Peak GPU Use (GiB)|2.74|22.37|3.56|4.79|\n\nWe also report memory consumption on KITTI in the table above. Each method uses the most Gaussians until no improvement can be made. We use a little bit more memory, but we manage to use more Gaussians to produce much better rendering.\n\n### **2. Impact of depth noise on the performance**\n\nAlthough Gaussians are fixed at depth with noise, Gaussian splatting is flexible enough to overfit the current frame and neighboring frames by tuning other attributes like color, opacity, and shape. Our results show that depth noises do not significantly impact the rendering performance. Meanwhile, we try to optimize the position of Gaussians along the ray direction, but we do not find an obvious improvement in rendering performance. We report additional results below. We also report a visual comparison using either fixed Gaussians or movable Gaussians (along the ray) at the link: https://imgur.com/a/oEfbgro.\n||10% pixels w/ noises|20% pixels w/ noises|30% pixels w/ noises|Gaussians movable along ray|Ours(w/o additional noises & fix)|\n|-|:-:|:-:|:-:|:-:|:-:|\n|PSNR$\\uparrow$|43.41|43.40|43.29|42.89|43.06|\n|SSIM$\\uparrow$|0.996|0.996|0.996|0.995|0.996|\n|LPIPS$\\downarrow$|0.015|0.015|0.015|0.020|0.013|\n\n### **3. Bundle adjustment**\n\nAs stated in Lines 243-245, our bundle adjustment is just used at the head frame in each section. Since the head is so important to start a section and is used as a reference by the following frame in the same section, this design stabilizes the optimization a lot and achieves high accuracy. Due to the large amount of Gaussians in total over a large-scale scene, we cannot conduct a final or global bundle adjustment using all Gaussians. The 80 iteration optimization is illustrated in Fig. 5. \n\n### **4. Numerical comparison in Tab. 9**\n\nWe confirmed the results are correct. We keep the experimental setting the same but just using different kinds of Gaussians. So, one analysis here is that these alternatives do not work well with some parameters, which produce rendering errors that can accumulate across frames fast in tracking.\n\n### **5. Code**\n\nThe code in the supplementary materials is merely for demonstration. We will release the code upon acceptance.", "claims_and_evidence": "The authors claim that by using view-tied Gaussian representation, the system reduces parameter storage (saving location, rotation, and variance) by directly binding each Gaussian to a depth pixel, and allows more Gaussians to represent local details. \nThe authors use ablation experiments on benchmarks to demonstrate the effectiveness of the method using ATE RMSE, PSNR, and other quantitative metrics.", "methods_and_evaluation_criteria": "Evaluation criteria include ATE RMSE, rendering quality, reconstruction results, and runtime and memory Usage. It is a good evaluation of various aspects of neural SLAM.", "theoretical_claims": "The contribution of this work lies more in algorithm design and is demonstrated experimentally rather than through formal proofs.\nThe main hypothesis is that \"view-tied\" Gaussians can represent geometry/color for local views (with supervision in depth) sufficiently without the expense of unconstrained 3D Gaussian parameters.", "experimental_designs_or_analyses": "The experimental metrics designed are good, but the benchmarks are all indoor datasets and do not involve extremely large scene mentioned in the abstract and introduction.", "relation_to_broader_scientific_literature": "The paper's approach builds on 3D Gaussian Splatting and Neural Implicit SLAM, and correctly cites the body of related research.", "essential_references_not_discussed": "No Essential References Not Discussed", "other_strengths_and_weaknesses": "1. Although the authors say their method works better for  extremely large scenes, most evaluations are still on single-room or short multi-room scales. So it would be more helpful if benchmark results on city scale or other large scenes could be shown.\n2. The author's method uses the depth from the RGB-D sensor for each frame. But depth is usually noisy, so why don't the author optimize the depth value based on the depth and the linear direction of the camera? Will this lead to better rendering results?\n3. For the Bundle Adjustment part, the authors did not explain how many iterations were used. And after the online mapping section, whether the author's method used final refine Bundle Adjustment for several iterations (similar to Mono-GS).", "other_comments_or_suggestions": "1. In the RMSE error section of Table 9, it seems that the results of the first three columns are filled in incorrectly, and are too different from the results of the fourth column.", "questions_for_authors": "see weekeness and other commonts", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kI9BIjzgPn", "reviewer": "Reviewer_wSTB", "summary": "The paper presents VTGaussian-SLAM, a novel RGBD SLAM system that utilizes view-tied 3D Gaussians for efficient mapping and tracking in large-scale scenes. It introduces the representation of Gaussians tied to depth pixels, thus improving optimization efficiency and reconstruction quality while enabling better scalability in SLAM applications.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and positive comments on our idea, contributions, evaluations, and supplementary materials.\n\n### **1. Impact of section length**\n\nFor fair comparisons with previous methods in rendering quality, we adopt the same number of iterations for mapping. But our Gaussians are view-tied, which are usually more than the ones in previous methods when covering the same number of frames. So, we set a proper section length to ensure our Gaussians in the same section can be well optimized in the same number of iterations. Comparisons in Tab. 10 indicate that the rendering may degenerate if more and more Gaussians are included in a section but still under the same number of iterations. We agree camera motion may also be a factor to consider, but we value more on the fair comparisons.\n\n### **2. Tracking and mapping iterations**\n\nFor fair comparisons, we follow the previous methods to conduct optimization in the same iterations on different benchmarks, such as 100 iterations for mapping and 60 iterations for tracking on Replica. Tab.11 reports the operation time on each frame. We will report these iterations clearly in our revision.\n\n### **3. Results on city-level scenes**\n\nWe follow previous methods like SplaTAM to report our evaluations on the widely used benchmarks such as ScanNet++. We also show our advantages in storage complexity, which can learn 20 times more Gaussians to recover more details on all frames than the latest methods, like SplaTAM, and also overcome the obstacle of out of memory at any frame during optimization. These advantages significantly improve our capability and performance in large-scale scenes. Due to the time limit, we only report our results on several city-level scenes in KITTI to evaluate tracking and mapping performance below. We also report a visual comparison of rendering performance at the link: https://imgur.com/a/FOIgcrz.\n|Methods|Gaussian-SLAM|SplaTAM|LoopSplat|Ours|\n|-|:-:|:-:|:-:|:-:|\n|00 (ATE RMSE$\\downarrow$[m])|3.02|58.83|2.22|**2.06**|\n|01 (ATE RMSE$\\downarrow$[m])|77.51|84.45|74.47|**29.01**|\n|05 (ATE RMSE$\\downarrow$[m])|128.88|80.39|117.43|**7.74**|\n|10 (ATE RMSE$\\downarrow$[m])|10.60|43.82|11.39|**4.54**|\n\n|Methods|Gaussian-SLAM|SplaTAM|LoopSplat|Ours|\n|-|:-:|:-:|:-:|:-:|\n|00 (PSNR$\\uparrow$)|15.51|9.82|15.82|**28.54**|\n|01 (PSNR$\\uparrow$)|15.95|12.89|14.69|**30.33**|\n|05 (PSNR$\\uparrow$)|16.22|26.48|15.98|**28.19**|\n|10 (PSNR$\\uparrow$)|15.58|25.58|14.58|**27.59**|\n|Peak GPU Use (GiB)|2.74|22.37|3.56|4.79|\n\nWe also report memory consumption on KITTI in the table above. Each method uses the most Gaussians until no improvement can be made. We use a little bit more memory, but we manage to use more Gaussians to produce much better rendering.\n\n### **4. Densify and prune Gaussians during mapping**\n\nNo, we do not need these operations, which can speed up the mapping. This is because our view-tied Gaussians provide adequate Gaussians to achieve better rendering quality and have an efficient strategy to resolve out-of-memory issues.\n\n### **5. Larger weight on the depth loss**\n\nAlthough depth maps from TUM-RGBD and ScanNet are not perfect, low qualities like motion blur, low resolution, and different exposure time make RGB supervision may be even more unreliable than depth to us. We found that weighting larger on depth maps can take full advantage of our view-tied Gaussians on real images.\n\n### **6. Minor issues**\n\nWe will revise the manuscript accordingly to resolve other minor issues.", "claims_and_evidence": "Yes, the claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, proposed methods and evaluation criteria make sense.", "theoretical_claims": "Yes, they are theoretically correct.", "experimental_designs_or_analyses": "Overall, the experimental designs are rigorous, and the discussions are comprehensive.", "relation_to_broader_scientific_literature": "The proposed methodology contributes to the improvement of GS-based RGBD SLAM systems.", "essential_references_not_discussed": "The most related works have been discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:   \n1)\tThe paper develops an out-of-core algorithm for 3D GS based SLAM.  \n2)\tThe paper proposes a simple yet effective supervision strategy to improve RGBD SLAM performance, achieving SOTA results.   \n3)\tThe paper is well written and easy to follow.   \nWeaknesses:  \n1)\tThe influence of section length. The length of the Gaussian section, i.e., N is an essential hyperparameters of the proposed system. The Gaussians are maintained within a fixed length of section, while the selection of this hyperparameter is sensitive to datasets. In Line 257-261, they mentioned they choose different N on different datasets, according to image resolution and mapping iterations. However, although the paper had conducted an ablation study on ScanNet to experimentally prove their selection of the length of section S is optimal, there are no substantive analysis and evidence to support this hyperparameters is related to image resolution and mapping iterations. Or should it be related to camera motion pattern? A deeper analysis is appropriated.  \n2)\tLack of implementation details. The number of optimization iterations in tracking and mapping appear not to be reported. Are the optimization iterations consistent across different datasets, and if not, how do they vary? Furthermore, Table 11 can include average operation time.  \n3)    Lack of evaluation on \"large-scale\" scenes. The paper emphasizes the scalability challenges of SLAM in large-scale scenes. However, the method is only evaluated on datasets with room-level scenes. In general, large-scale scenes in SLAM tasks typically involve building- or even city-level environments. A more comprehensive evaluation on such large-scale scenarios, e.g., KITTI, would strengthen the paper's claims regarding scalability.", "other_comments_or_suggestions": "1)\tOn the top of each subfigure in Figure 1, is it Section 1 or Section o, {g}$^1$ or  {g}$^o$\n2)\tLine 369, “Without using sections (“1”), we cannot ...”, is it section (“1”) or section (\"o”).\n3)\tFigure 4 can include the corresponding rendering image and error for each section to enhance visualization.", "questions_for_authors": "1)\tDid you prune or densify the GS during the mapping iterations?\n2)\tSelection of hyperparameters {α, β} in tracking. Depth maps from TUM-RGBD and ScanNet++ exhibit significantly higher levels of noise compared to those from Replica. When depth information is imperfect, it is intuitive that the system should rely less on depth loss. This suggests that the value of β (the weight assigned to depth loss) should be smaller in such scenarios. However, the system uses a much larger β when running on TUM-RGBD and ScanNet. Could you provide an explanation for this choice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dysRfal5nK", "reviewer": "Reviewer_HsjG", "summary": "- This paper addresses the limitation of traditional 3DGS-SLAM methods, which struggle to scale up to extremely large scenes due to inefficient tracking and mapping strategies. \n- The authors propose tracking and mapping strategies based on a new 3D representation called view-tied 3D Gaussians, which simplifies traditional 3D Gaussians by tying them to depth pixels, eliminating the need to learn their locations, rotations, and multi-dimensional variances. \n- The proposed method demonstrates advantages in both reducing storage requirements and improving rendering quality. Extensive experiments supported the effectiveness of their design.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and positive comments on motivation and performance.\n### **1. Results on city-level scenes**\nFollowing SplaTAM, we evaluate on the widely used benchmarks such as ScanNet++ and demonstrate superior storage efficiency, learning 20 times more Gaussians for more detailed rendering while overcoming out of memory issues during optimization. These advantages significantly improve our capability in large-scale scenes. Due to the time limit, We additionally report our results on several city-level scenes in KITTI below, and visual comparison is shown at https://imgur.com/a/FOIgcrz.\n|ATE RMSE$\\downarrow$[m]|Gaussian-SLAM|SplaTAM|LoopSplat|Ours|\n|-|:-:|:-:|:-:|:-:|\n|00|3.02|58.83|2.22|**2.06**|\n|01|77.51|84.45|74.47|**29.01**|\n|05|128.88|80.39|117.43|**7.74**|\n|10|10.60|43.82|11.39|**4.54**|\n\n|PSNR$\\uparrow$|Gaussian-SLAM|SplaTAM|LoopSplat|Ours|\n|-|:-:|:-:|:-:|:-:|\n|00|15.51|9.82|15.82|**28.54**|\n|01|15.95|12.89|14.69|**30.33**|\n|05|16.22|26.48|15.98|**28.19**|\n|10|15.58|25.58|14.58|**27.59**|\n|Peak GPU Use (GiB)|2.74|22.37|3.56|4.79|\n\nMemory on KITTI is in the table above. Each method is using the most Gaussians until no improvement can be made. We use more memory, but manage to produce much better rendering with more Gaussians.\n\nWe will also cite GO-SLAM. With the same RGBD setting as ours, GO-SLAM did not report results on city-level scenes either, and we cannot report its plausible results on KITTI at this moment. Note that it requires a RGB or stereo setting for larger scenes. \n### **2. Novelty and effectiveness**\nOur novelty not only lies in the view-tied Gaussians, which save storage for more Gaussians to recover more details, but also novel tracking and mapping strategies to work with view-tied Gaussians.\n\nOur tracking strategy resolves the camera pose error accumulation when not accessing all Gaussians as a global reference. Our novel mapping strategy finds a balance between storage complexity and rendering quality. We believe these novel strategies are not a kind of engineering job. Their effectiveness has been justified in our extensive experiments and ablation studies.\n### **3. The 1st misunderstanding**\n“The View-Tied Gaussians approach appears to … current frame in prior works.” is a misunderstanding. As our response to your question 2 and explanations in Lines 214-239, we keep all Gaussians in the current section, which covers the current frame and its neighbor frames, learnable, rather than merely focusing on the current frame. This differentiates our method from the incremental SLAM framework.\n### **4. Runtime**\nCompared to classic SLAM methods, there is still a large room for rendering-based SLAM methods to improve the runtime efficiency. However, rendering-based SLAM methods like ours provide the capability of novel view synthesis, which can be directly used in VR. This is a vital function that classic SLAM methods cannot provide. Tab. 11 reports our runtime efficiency is comparable to the latest rendering-based methods.\n### **5. Difference to Point-SLAM**\nAlthough we are using a simplified isotropic Gaussian representation, each sphere is still a Gaussian rather than a point, including other attributes. So, ours is a GS-based SLAM, which is different from NeRF-based SLAM like Point-SLAM. Runtime comparisons in Tab. 11 show splatting is faster than ray tracing, and numerical comparisons in Tab. 3-8 show our superior performance over Point-SLAM.\n### **6. Global bundle adjustment**\nAs stated in Sec. 3.5, we do have BA, but only at the head frame. Since we employ a large amount of Gaussians over the scene, and cannot access all Gaussians at the same time, we cannot perform a global BA over all frames. But our superior performance shows no degeneration.\n\nBut we do have special designs to ensure global consistency and optimum with multi-view constraints in both tracking and mapping, like finding the overlapping section in the most front as a common reference in tracking and along with fixed Gaussians in other sections when mapping the most current section.\n### **7. Difference to traditional GS-based SLAM**\nFirstly, our Gaussians are fixed at depth points and have fewer attributes to learn. Secondly, we only need to keep Gaussians in the current section learnable and in the memory without using keyframes. While traditional GS-based SLAM methods require keeping all Gaussians learnable in the memory to maintain consistency to the current frame and all keyframes. These designs enable us to employ more Gaussians to recover more details in larger scenes.\n### **8. The 2nd misunderstanding**\n“In traditional Gaussian splatting, … involved in optimization” is a misunderstanding. With a background of SLAM, traditional GS-based methods require keeping all Gaussians (rather than merely the ones in the current frame) learnable and getting involved in the optimization, since they need to constrain Gaussians using both the current view and the keyframes, aiming to maintain the global consistency.", "claims_and_evidence": "Yes. The claim that \"existing methods struggle to scale up to extremely large scenes due to inefficient tracking and mapping strategies\" is generally valid. However, the paper doesn't provide sufficient experimental evidence to demonstrate the superiority of VTGaussian-SLAM in **large-scale scenarios**. The experiments are conducted only on small-scale scenes from datasets such as Replica, TUM-RGBD, and small subset of ScanNet++, which do not adequately test the scalability of the proposed method. To convincingly support the claim, the authors should evaluate their approach on larger-scale scenes. A relevant benchmark for comparison could be GO-SLAM: Global Optimization for Consistent 3D Instant Reconstruction (ICCV 2023), which addresses similar challenges in large-scale environments.", "methods_and_evaluation_criteria": "For method: the proposed View-Tied 3D Gaussians method introduces a novel approach to 3D representation in SLAM systems. By tying a 3D Gaussian to each pixel in the depth map, the positions of these Gaussians are determined solely by depth and camera poses, eliminating the need to learn and store their locations or perform density control. This design significantly reduces memory and computational overhead. Additionally, the proposed tracking and mapping strategies focus on rendering and optimizing only a subset of Gaussians associated with the most recent views, rather than all Gaussians in the scene. This approach removes the need to maintain and optimize all Gaussians in memory throughout the training process, thereby improving the scalability of 3D Gaussian Splatting (3DGS) in SLAM applications.\n\nFor evaluation: As I suggested in the previous section, the experiments are insufficient. Only small-scale datasets such as Replica, TUM-RGBD, and small subsets of ScanNet++ are conducted, which do not adequately test the scalability of the proposed method. To convincingly support the claim, the authors should evaluate their approach on larger-scale scenes.", "theoretical_claims": "Yes. The theoretical proofs in this paper are quite concise, as the work is primarily based on modifications to an existing code framework. Consequently, the tracking and mapping sections lack substantial theoretical development, and the focus of the paper is predominantly on the implementation aspects rather than theoretical contributions.", "experimental_designs_or_analyses": "Yes. I review the experiments in 4.1. Comparisons and 4.2. Ablation Studies and Analysis. The main experimental results are compelling. The authors demonstrate that their view-tied Gaussians significantly reduce storage requirements, enabling the maintenance of a large number of Gaussians within limited GPU memory. Furthermore, the ablation studies provide additional validation for the effectiveness of the proposed VT (view-tied) approach. Specifically, the ablation study compares different attributes of 3D Gaussians, including anisotropic Gaussians (aniso), isotropic Gaussians (iso), and view-tied Gaussians (VT), further confirming the advantages of the VT design.", "relation_to_broader_scientific_literature": "The key contributions of this paper build upon and address limitations in recent works such as (Keetha et al., 2024; Matsuki et al., 2024; Huang et al., 2024b; Yan et al., 2024; Yugay et al., 2023; Sandström et al., 2024). which employ various tracking and mapping strategies, but requires maintaining and optimizing all Gaussians covering the scene within limited GPU memory to ensure color and geometry consistency across all previous views. \n\nHere are also some more complex systems (Liso et al., 2024; Zhu et al., 2024; Bruns et al., 2024; Sandström et al., 2024), that incorporate loop closure mechanisms into their optimization processes. However, detecting loop closures across views often relies on pre-trained priors and is highly sensitive to image quality.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Weakness:\n- I believe the novelty and effectiveness of this paper are somewhat weak, although I agree with the motivation of addressing the scalability limitations of existing methods that struggle to optimize all 3D Gaussians within limited GPU memory. The proposed Learnable View-Tied Gaussians and Frozen View-Tied Gaussians essentially follow a standard incremental SLAM framework. The View-Tied Gaussians approach appears to focus primarily on optimizing the 3D scene for the current view, which seems conceptually similar to the tracking and mapping of the current frame in prior works. As such, the contribution leans more toward engineering implementation rather than groundbreaking innovation.\n- The experiments fail to demonstrate the superiority of the proposed method in large-scale scenarios. The paper and video does not provide sufficient evidence regarding key performance metrics such as tracking and mapping accuracy, rendering quality, memory efficiency, and computational time in large-scale environments. While the results on smaller datasets (e.g., Replica, TUM-RGBD, and ScanNet++) are promising, they do not adequately validate the scalability claims. To strengthen the paper, the authors should include evaluations on larger-scale benchmarks, which would more convincingly showcase the advantages of their approach in real-world applications.\n- The proposed method cannot achieve real-time performance with approximately 0.5 tracking FPS in the easiest Replica dataset, which is unacceptable for the SLAM framework with real-time demands.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. The paper proposes simplifying an ellipsoid Gaussian into a sphere, which retains only a color  c ∈ R1×3, a radius (variance) r∈R1\n, and an opacity o∈R1. How does this simplification differ from the approach in Point-SLAM: Dense Neural Point Cloud-based SLAM (ICCV 2023)? It seems that such a simplified Gaussian representation effectively reduces to a Neural Point Cloud. Could the authors clarify the distinctions and advantages of their method compared to this prior work?\n2. The strategy of selecting overlapping regions and Frozen View-Tied Gaussians raises concerns about the ability to perform global bundle adjustment (BA). Could this approach lead to a degradation in the accuracy of the map and camera poses, potentially resulting in suboptimal solutions? Does the authors' framework include additional mechanisms to ensure global consistency and maintain optimality in the reconstruction?\n3. I am unclear about the concept of View-tied Gaussians. In traditional Gaussian splatting, only the Gaussians projected onto the current camera view are involved in optimization. How does View-tied Gaussians fundamentally differ from this approach? Could the authors elaborate on the unique aspects and advantages of their method compared to the standard Gaussian splatting framework?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2F9wHqJiS1", "reviewer": "Reviewer_N2ss", "summary": "To address a high memory consumption issue of 3DGS, this paper proposes view-tied 3DGS, which determines Gaussians based on the views.\nThe Gaussians from the last frame are tracked and processed in sections. Since the method is view-dependent, it efficiently reduces the storage required for location, rotation, and scale values. The location of Gaussians is expressed through depth, and adjacent frames are used to handle missing depth information.\nWhen a new section is optimized, pose optimization is performed using rendering loss. To minimize total error, the head frame of the previous section is used. The Gaussians in the scene are trained using L1, SSIM and depth loss. Finally, BA is applied by optimizing the Gaussians of the head frame and camera pose to reduce absolute pose error.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and positive comments on our idea, contributions, evaluations, and supplementary materials.\n\n### **1. Overview**\nWe will revise the overview accordingly to make it easier to follow.\n\n### **2. Runtime comparison in Tab.11**\nAs stated in Lines 434-438 left, we manage to optimize many more Gaussians at each frame for much better rendering quality and more accurate pose estimation than other NeRF-based and GS-based methods, which sacrifices a little bit more runtime in total. However, we show a great advantage if we compare the runtime averaged on each Gaussian. All GS-based methods use the most Gaussians until no improvement can be made further in Tab. 11. We report peak GPU memory usage as a supplement to Tab. 11 below.\n|Methods|NICE-SLAM|Point-SLAM|SplaTAM|Gaussian-SLAM|Ours|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Peak GPU Use (GiB)|12.0|7.7|18.5|4.2|5.4|\n\n\nAlthough Gasussian-SLAM uses fewer memory, we employ more Gaussians and produce much better rendering, as shown in the following. We also report additional experiments on city-level scenes. Due to the time limit, we only report our results on several scenes in KITTI to evaluate tracking and mapping performance below. We also provide a visual comparison of rendering performance at the link: https://imgur.com/a/FOIgcrz.\n|Methods|Gaussian-SLAM|SplaTAM|LoopSplat|Ours|\n|-|:-:|:-:|:-:|:-:|\n|00 (ATE RMSE$\\downarrow$[m])|3.02|58.83|2.22|**2.06**|\n|01 (ATE RMSE$\\downarrow$[m])|77.51|84.45|74.47|**29.01**|\n|05 (ATE RMSE$\\downarrow$[m])|128.88|80.39|117.43|**7.74**|\n|10 (ATE RMSE$\\downarrow$[m])|10.60|43.82|11.39|**4.54**|\n\n|Methods|Gaussian-SLAM|SplaTAM|LoopSplat|Ours|\n|-|:-:|:-:|:-:|:-:|\n|00 (PSNR$\\uparrow$)|15.51|9.82|15.82|**28.54**|\n|01 (PSNR$\\uparrow$)|15.95|12.89|14.69|**30.33**|\n|05 (PSNR$\\uparrow$)|16.22|26.48|15.98|**28.19**|\n|10 (PSNR$\\uparrow$)|15.58|25.58|14.58|**27.59**|\n|Peak GPU Use (GiB)|2.74|22.37|3.56|4.79|\n\n\n### **3. Time on overlapping section selection**\nTo determine the overlapping section selection, we project the downsampled depth points (200000) in the current view to the previous candidates. This simple manner is quite fast, and does not bring a burden, as shown in the time comparison below.\n|Number of Overlapping Candidate Frames|200|400|800|1600|\n|-|:-:|:-:|:-:|:-:|\n|Time (s)|0.038|0.079|0.144|0.282|\n\n### **4. What if depth is not available**\nActually, SLAM methods using RGBD images are dedicated to indoor scenes. For outdoor scenes, we may not have sensor-captured depth maps, but we can use monocular depth priors, such as DepthAnything or MiDaS, to predict depth maps from RGB images as initialization. Then, in the first several frames, we can register the depth to the sparse depth from SfM (Structure from Motion). In the following frames, we can register the predicted depth to the depth rendered with Gaussians as a coarse depth to initialize Gaussians.\n\n### **5. Memory comparisons**\nIn Tab.11, we report the memory cost with the most Gaussians in each method until no improvement can be made. Please see our memory consumption on KITTI in the second table in our response to the question 2 above.\n\n### **6. Impact of depth noise on the performance**\nOur results on real datasets like Scannet were reported using depth maps with noise. \n\nAlthough our Gaussians are fixed at depth with noise, Gaussian splatting is flexible enough to overfit the current frame and neighboring frames by tuning other attributes like color, opacity, and shape. Our results show that depth noise does not impact the rendering. We report additional results below.\n\n||10% pixels w/ noises|20% pixels w/ noises|30% pixels w/ noises|Ours(w/o additional noises & fix)|\n|-|:-:|:-:|:-:|:-:|\n|PSNR$\\uparrow$|43.41|43.40|43.29|43.06|\n|SSIM$\\uparrow$|0.996|0.996|0.996|0.996|\n|LPIPS$\\downarrow$|0.015|0.015|0.015|0.013|\n\n### **7. Minor issues**\nWe will revise the manuscript accordingly to resolve other minor issues.", "claims_and_evidence": "This paper's main claim is that instead of storing all 3DGS parameters, it proposes a section-based storage approach, making it more efficient on limited GPU resources. The paper argues that the model can operate on large-scale scenes, demonstrating better rendering performance and lower pose error compared to existing models.\nHowever, the experiments are conducted only on indoor datasets, and no actual large-scale scenes are presented.", "methods_and_evaluation_criteria": "Instead of representing the entire scene as Gaussians, this paper proposes an approach based on views, and the scene is represented using depth values from each view.\nTo reduce the number of parameters, location is replaced with depth, and rotation is eliminated by representing Gaussians with a sphere shape. Within each section, Gaussians are merged to create a scene. To further reduce redundancy, visibility across view is measured, and only Gaussians in invisible regions are additionally stored.\nFor each new section, the head frame from the previous section, where views have the highest overlap, is selected. The Gaussians are then stored based on these views.\n\nThe comparison models include Neural Implicit Fields models and 3DGS-based models.\nOn the Replica dataset, D-L1 and F1 score were measured. On other datasets, ATE RMSE was used for tracking comparisons, and PSNR, SSIM, and LPIPS were used for rendering comparisons.", "theoretical_claims": "There are no specifically proposed theoretical claims.", "experimental_designs_or_analyses": "The experiments follow the other SLAM methodologies and are conducted on the Replica, TUM-RGBD, ScanNet and ScanNet++ datasets. The evaluations are performed on an NVIDIA RTX 4090, where comparisons of runtime and the number of Gaussians are presented. Additionally, the experiments include comparisons between anisotropic and isotropic Gaussians, analyses of different section lengths, evaluations of overlap selection methods, and ablation studies on the impact of the visibility mask.", "relation_to_broader_scientific_literature": "The paper employs a method that represents RGBD SLAM using Gaussians and proposes a section-based representation. Instead of using point clouds, it adopts a 3DGS primitive-based approach. However, since the Gaussians are tied to depth-based representation, extending this method to RGB SLAM appears challenging.", "essential_references_not_discussed": "There do not appear to be any essential references that are not discussed.", "other_strengths_and_weaknesses": "**Strengths**\n\nIt is interesting that the model achieves good rendering results with a more simplified representation compared to other models.\n\n**Weaknesses**\n\nThe explanation of the overlap section in the method is not smooth, and the description of the threshold $\\gamma$ is unclear. Despite the significant impact of the overlap method, it is not well-explained in the overview, making it difficult to understand.\n\nThere are no experimental results on actual large-scale scenes. In Table 11, the runtime is slower compared to G-SL and NICE, and both the total and maximum number of Gaussians are higher than in G-SL. This suggests that additional experiments on large-scale scenes and a direct comparison of GPU memory usage are necessary.\n\nAs the number of frames increases, the number of overlap candidates is likely to grow, potentially increasing the time required to search for sections. However, there are no experiments addressing this issue. Additionally, as the number of sections increases, the redundancy of Gaussians stored in the head frame is expected to rise. \n\nSince the model is designed based on depth modality, it is likely to be highly dependent on dataset-specific factors such as the number of captured frames and resolution. Furthermore, in large-scale outdoor environments where dense depth acquisition is not feasible, the model may fail to operate effectively. This suggests that the model has generalization limitations.", "other_comments_or_suggestions": "There is a spacing typo on line 417: \"Visibility .\"", "questions_for_authors": "How much is the actual difference in memory usage in Table 11?\n\nIn Supplementary Section D.1, I am curious about the model's robustness to input depth noise. Are experimental results available to assess this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces VTGaussian-SLAM, an RGB-D SLAM system that utilizes a novel view-tied 3D Gaussian representation. By anchoring Gaussians to depth pixels, the method avoids the need to optimize explicit parameters like position, rotation, and scale, significantly reducing memory usage and computational overhead. The system organizes frames into sections, optimizing only the current and overlapping sections' Gaussians at any given time, enhancing scalability and efficiency. 

All reviewers learned to be positive after reading comprehensive feedback from the authors. Reviewers 4tKp and wSTB gave the acceptance score, and reviewers HsjG and N2ss gave a weak acceptance score. AC carefully read the review and agreed with the reviewer's recommendation. AC recommends applying the additional results and discussions in the revised version.
Decision: Accept (poster)