Splat-SLAM: Globally Optimized RGB-only SLAM with 3D Gaussians

## Abstract

Abstract 3D Gaussian Splatting has emerged as a powerful representation of geometry and appearance for RGB-only dense Simultaneous Localization and Mapping (SLAM), as it provides a compact dense map representation while enabling efficient and high-quality map rendering.
However, existing methods show significantly worse reconstruction quality than competing methods using other 3D representations, e.g. neural points clouds, since they either do not employ global map and pose optimization or make use of monocular depth.
In response, we propose the first RGB-only SLAM system with a dense 3D Gaussian map representation that utilizes all benefits of globally optimized tracking by adapting dynamically to keyframe pose and depth updates by actively deforming the 3D Gaussian map. Moreover, we find that refining the depth updates in inaccurate areas with a monocular depth estimator further improves the accuracy of the 3D reconstruction. Our experiments on the Replica, TUM-RGBD, and ScanNet datasets indicate the effectiveness of globally optimized 3D Gaussians, as the approach achieves superior or on par performance with existing RGB-only SLAM methods methods in tracking, mapping and rendering accuracy while yielding small map sizes and fast runtimes. The source code is available at https://github.com/eriksandstroem/Splat-SLAM .

## 1 Introduction

A common factor within the recent trend of dense SLAM is that the majority of works reconstruct a dense map by optimizing a neural implicit encoding of the scene, either as weights of an MLP [, , , ], as features anchored in dense grids [, , , , , , , , ], using hierarchical octrees [], via voxel hashing [, , , , ], point clouds [, , , ] or axis-aligned feature planes [, ]. We have also seen the introduction of 3D Gaussian Splatting (3DGS) to the dense SLAM field [, , , , ].

Out of this 3D representation race there is, however, not yet a clear winner. In the context of dense SLAM, a careful modeling choice needs to be made to achieve accurate surface reconstruction as well as low tracking errors. Some takeaways can be deduced from the literature: neural implicit point cloud representations achieve state-of-the-art reconstruction accuracy [, , ], especially with RGBD input. At the same time, 3D Gaussian splatting methods yield the highest fidelity renderings [, , , , ] and show promise in the RGB-only setting due to their flexibility in optimizing the surface location [, ]. However, they are not leveraging any multi-view depth or geometric prior leading to poor geometry in the RGB-only setting. The majority of the aforementioned works only deploy so called frame-to-model tracking, and do not implement global trajectory and map optimization, leading to excessive drift, especially in real world conditions. Instead, to this date, frame-to-frame tracking methods, coupled with loop closure and global bundle adjustment (BA) achieve state-of-the-art tracking accuracy [, , ]. However, they either use hierarchical feature grids [, ], not suitable for map deformations at *e.g.* loop closure as they require expensive reintegration strategies, or neural point clouds as in GlORIE-SLAM []. While the neural point cloud is straightforward to deform, the depth guided rendering leads to artifacts when the depth is noisy and the surface estimation can only be adjusted locally since the point locations are not optimized directly.

In this work we propose an RGB-only SLAM system that combines the strengths of frame-to-frame tracking using recurrent dense optical flow [] with the fidelity of 3D Gaussians as the map representation [] (see fig. 1). The point-based 3D Gaussian map enables online map deformations at loop closure and global BA. To enable accurate surface reconstruction, we leverage consistent so called proxy depth that combines multi-view depth estimation with learned monocular depth.

Our contribution comprises, for the first time, a SLAM pipeline encompassing all the following parts:

- $\bullet$
A frame-to-frame RGB-only tracker with global consistency.
- $\bullet$
A dense deformable 3D Gaussian map that adapts online to loop closure and global BA.
- $\bullet$
A proxy depth map consisting of on-the-fly optimized multi-view depth and a monocular depth estimator leading to improved rendering and reconstruction quality.
- $\bullet$
Improved map sizes and runtimes compared to other dense SLAM approaches.

Figure: Figure 1: Splat-SLAM. Our system yields accurate scene reconstruction (rendering depth L1) and rendering (PSNR) and on par tracking accuracy (ATE RMSE) to GlORIE-SLAM and map size to MonoGS. The results averaged over all keyframes. The scene is from TUM-RGBD [] fr1 room.
Refer to caption: extracted/5621324/figs/teaser/gt_teaser.png

## 2 Related Work

Dense Visual SLAM.
Curless and Levoy [] pioneered dense online 3D mapping with truncated signed distance functions, with KinectFusion [] demonstrating real-time SLAM via depth maps.
Enhancements like voxel hashing [, , , , ] and octrees [, , , , ] improved scalability, while point-based SLAM [, , , , , , , , , ] has also been effective. To address pose drift, globally consistent pose estimation and dense mapping techniques have been developed, often dividing the global map into submaps [, , , , , , , , , , , , , , , , ]. Loop detection triggers submap deformation via pose graph optimization [, , , , , , , , , , , , , , , , , , , ]. Sometimes global BA is used for refinement [, , , , , , , , , ]. 3D Gaussian SLAM with RGBD input has also been shown, but these methods do not consider global consistency via *e.g.* loop closure [, , ]. Other approaches to global consistency minimize reprojection errors directly, with DROID-SLAM [] refining dense optical flow and camera poses iteratively, and recent enhancements like GO-SLAM [], HI-SLAM [], and GlORIE-SLAM [] optimizing factor graphs for accurate tracking. For a recent survey on NeRF-inspired dense SLAM, see [].

RGB-only Dense Visual SLAM.
The majority of NeRF inspired dense SLAM works using only RGB cameras do not address the problem of global map consistency or requires expensive reintegration strategies via backpropagation [, , , , , , , , , ]. Instead, the concurrent GlORIE-SLAM [] uses a feature based point cloud which can adapt to global map changes in a straight forward way. However, redundant points are not pruned, leading to large map sizes. Furthermore, the depth guided sampling during rendering leads to rendering artifacts when noise is present in the estimated depth. MonoGS [] and Photo-SLAM [] pioneered RGB-only SLAM with 3D Gaussians. However, they lack proxy depth which prevents them from achieving high accuracy mapping. MonoGS [] also lacks global consistency. Concurrent to our work, MoD-SLAM [] uses an MLP to parameterize the map via a unique reparameterization.

Figure: Figure 2: Splat-SLAM Architecture. Given an RGB input stream, we track and map each keyframe, initially estimating poses through local bundle adjustment (BA) using a DSPO (Disparity, Scale and Pose Optimization) layer. This layer integrates pose and depth estimation, enhancing depth with monocular depth. It further refines poses globally via online loop closure and global BA. The proxy depth map merges keyframe depths $\tilde{D}$ from the tracking with monocular depth $D^{mono}$ to fill gaps. Mapping employs a deformable 3D Gaussian map, optimizing its parameters through a re-rendering loss. Notably, the 3D map adjusts for global pose and depth updates before each mapping phase.
Refer to caption: x1.png

## 3 Method

Splat-SLAM is a monocular SLAM system which tracks the camera pose while reconstructing the dense geometry of the scene in an online manner. This is achieved through the following steps: We first track the camera by performing local BA on selected keyframes by fitting them to dense optical flow estimates. The local BA optimizes the camera pose as well as the dense depth of the keyframe. For global consistency, when loop closure is detected, loop BA is performed on an extended graph including the loop nodes and edges (section 3.1). Interleaved with tracking, mapping is done on a progressively growing 3D Gaussian map which deforms online to the keyframe poses and so called proxy depth maps (section 3.2). For an overview of our method, see fig. 2.

### 3.1 Tracking

To predict the motion of the camera during scene exploration, we use a pretrained recurrent optical flow model [] coupled with a Disparity, Scale and Pose Optimization (DSPO) layer [] to jointly optimize camera poses and per pixel disparities.
In the following, we describe this process in detail.

Optimization is done with the Gauss-Newton algorithm over a factor graph $G(V,E)$, where the nodes $V$ store the keyframe pose and disparity, and edges $E$ store the optical flow between keyframes. Odometry keyframe edges are added to $G$ by computing the optical flow to the last added keyframe. If the mean flow is larger than a threshold $\tau\in\mathbb{R}$, the new keyframe is added to $G$. Edges for loop closure and global BA are discussed later. Importantly, the same objective is optimized for local BA, loop closure and global BA, but over factor graphs with different structures.

The DSPO layer consists of two optimization objectives that are optimized alternatively. The first objective, typically termed Dense Bundle Adjustment (DBA) [] optimizes the pose and disparity of the keyframes jointly, eq. 1. Specifically, the objective is optimized over a local graph defined within a sliding window over the current frame.

$$ $\mathop{\arg\min}_{\omega,d}\sum_{(i,j)\in E}\left\|\tilde{p}_{ij}-K\omega_{j} ^{-1}(\omega_{i}(1/d_{i})K^{-1}[p_{i},1]^{T})\right\|^{2}_{\Sigma_{ij}}\enspace,$ (1) $$

with $\tilde{p}_{ij}\in\mathbb{R}^{(W\times H\times 2)\times 1}$ being the flattened predicted pixel coordinates when the pixels $p_{i}\in\mathbb{R}^{(W\times H\times 2)\times 1}$ from keyframe $i$ are projected into keyframe $j$ using optical flow. Further, $K$ is the camera intrinsics, $\omega_{j}$ and $\omega_{i}$ the camera-to-world extrinsics for keyframes $j$ and $i$, $d_{i}$ the disparity of pixel $p_{i}$ and $\|\cdot\|_{\Sigma_{ij}}$ is the Mahalanobis distance with diagonal weighting matrix $\Sigma_{ij}$. Each weight denotes the confidence of the optical flow prediction for each pixel in $\tilde{p}_{ij}$.
For clarity of the presentation, we omit homogeneous coordinates.

The second objective introduces monocular depth $D^{\text{mono}}$ as two additional data terms. The monocular depth $D^{\text{mono}}$ is predicted at runtime by a pretrained relative depth DPT model [].

$$ $\displaystyle\mathop{\arg\min}_{d^{h},\theta,\gamma}$ $\displaystyle\sum_{(i,j)\in E}\left\|\tilde{p}_{ij}-K\omega_{j}^{-1}(\omega_{i }(1/d_{i}^{h})K^{-1}[p_{i},1]^{T})\right\|^{2}_{\Sigma_{ij}}$ (2) $\displaystyle+$ $\displaystyle\alpha_{1}\sum_{i\in V}\left\|d_{i}^{h}-\left(\theta_{i}(1/D^{ \text{mono}}_{i})+\gamma_{i}\right)\right\|^{2}+\alpha_{2}\sum_{i\in V}\left\| d_{i}^{l}-\left(\theta_{i}(1/D^{\text{mono}}_{i})+\gamma_{i}\right)\right\|^{2 }\enspace.$ $$

Here, the optimizable parameters are the scales $\theta\in\mathbb{R}$, shifts $\gamma\in\mathbb{R}$ and a subset of the disparities $d^{h}$ classified as being high error (explained later). This is done since the monocular depth is only deemed useful where the multi-view disparity $d_{i}$ optimization is inaccurate. Furthermore, $\alpha_{1}\!<\!\alpha_{2}$, which is done to ensure that the scales $\theta$ and shifts $\gamma$ are optimized with the preserved low error disparities $d^{l}$. The scale $\theta_{i}$ and shift $\gamma_{i}$ are initialized using least squares fitting

$$ $\{\theta_{i},\,\gamma_{i}\}=\mathop{\arg\min}_{\theta,\gamma}\sum_{(u,v)}\Big{ (}\big{(}\theta(1/D^{\text{mono}}_{i})+\gamma\big{)}-d_{i}^{l}\Big{)}^{2}\enspace.$ (3) $$

Equation 1 and eq. 2 are optimized alternatingly to avoid the scale ambiguity encountered if $d$, $\theta$, $\gamma$ and $\omega$ are optimized jointly.

Next, we describe how high and low error disparities are classified. For a given disparity map $d_{i}$ (separated into low and high error parts $\{d_{i}^{l},d_{i}^{h}\}$) for frame $i$, we denote the corresponding depth $\tilde{D}_{i}=1/d_{i}$. Pixel correspondences $(u,v)$ and $(\hat{u},\hat{v})$ between keyframes $i$ and $j$ respectively are established by warping $(u,v)$ into frame $j$ with depth $\tilde{D}_{i}$ as

$$ $p_{i}=\omega_{i}\tilde{D}_{i}(u,v)K^{-1}[u,v,1]^{T},\qquad[\hat{u},\hat{v},1]^ {T}\propto K\omega_{j}^{-1}[p_{i},1]^{T}\enspace.$ (4) $$

The corresponding 3D point to $(\hat{u},\hat{v})$ is computed from the depth at $(\hat{u},\hat{v})$ as

$$ $p_{j}=\omega_{j}\tilde{D}_{j}(\hat{u},\hat{v})K^{-1}[\hat{u},\hat{v},1]^{T}\enspace.$ (5) $$

If the L2 distance between $p_{i}$ and $p_{j}$ is smaller than a threshold, the depth $\tilde{D}_{i}(u,v)$ is consistent between $i$ and $j$. By looping over all keyframes except $i$, the global two-view consistency $n_{i}$ can be computed for frame $i$ as

$$ $n_{i}(u,v)=\sum_{\begin{subarray}{c}k\in\text{KFs},\\ k\neq i\end{subarray}}\mathds{1}\Big{(}\left\|p_{i}-p_{k}\right\|_{2}<\eta \cdot\text{average}(\tilde{D}_{i})\Big{)}\enspace.$ (6) $$

Here, $\mathds{1}(\cdot)$ is the indicator function and $\eta\in\mathbb{R}_{\geq 0}$ is a hyperparameter and $n_{i}$ is the total two-view consistency for pixel $(u,v)$ in keyframe $i$. $\tilde{D}_{i}(u,v)$ is valid if $n_{i}$ is larger than a threshold.

Loop Closure. To mitigate scale and pose drift, we incorporate loop closure along with online global bundle adjustment (BA) in addition to local window frame tracking. Loop detection is achieved by calculating the mean optical flow magnitude between the current active keyframes (within the local window) and all previous keyframes. Two criteria are evaluated for each keyframe pair:
First, the optical flow must be below a specified threshold $\tau_{\text{loop}}$, ensuring sufficient co-visibility between the views.
Second, the time interval between the frames must exceed a predefined threshold $\tau_{t}$ to prevent the introduction of redundant edges into the graph. When both criteria are met, a unidirectional edge is added to the graph. During the loop closure optimization process, only the active keyframes and their connected loop nodes are optimized to keep the computational load manageable.

Global BA. For the online global BA, a separate graph that includes all keyframes up to the present is constructed. Edges are introduced based on the temporal and spatial relationships between the keyframes, as outlined in []. Following the approach detailed in [], we execute online global BA every 20 keyframes. To maintain numerical stability, the scales of the disparities and poses are normalized prior to each global BA optimization. This normalization involves calculating the average disparity $\bar{d}$ across all keyframes and then adjusting the disparity to $d_{norm}=d/\bar{d}$ and the pose translation to $t_{norm}=\bar{d}t$.

### 3.2 Deformable 3D Gaussian Scene Representation

We adopt a 3D Gaussian Splatting representation [] which deforms under DSPO or loop closure optimizations to achieve global consistency.
Thus, the scene is represented by a set $\mathcal{G}=\{g_{i}\}_{i=1}^{N}$ of 3D Gaussians. Each Gaussian primitive $g_{i}$, is parameterized by a covariance matrix $\Sigma_{i}\in\mathbb{R}^{3\times 3}$, a mean $\bm{\mu}_{i}\in\mathbb{R}^{3}$, opacity $o_{i}\in[0,1]$, and color $\mathbf{c}_{i}\in\mathbb{R}^{3}$. All attributes of each Gaussian are optimized through back-propagation. The density function of a single Gaussian is described as

$$ $g_{i}(\mathbf{x})=\exp\Big{(}-\frac{1}{2}(\mathbf{x}-\bm{\mu}_{i})^{\top} \Sigma_{i}^{-1}(\mathbf{x}-\bm{\mu}_{i})\Big{)}\enspace.$ (7) $$

Here, the spatial covariance $\Sigma_{i}$ defines an ellipsoid and is decomposed as $\Sigma_{i}=R_{i}S_{i}S^{T}_{i}R^{T}_{i}$, where $S_{i}=\mathrm{diag}(s_{i})\in\mathbb{R}^{3\times 3}$ is the spatial scale and $R_{i}\in\mathbb{R}^{3\times 3}$ represents the rotation.

Rendering. Rendering color and depth from $\mathcal{G}$, given a camera pose, involves first projecting (known as “splatting”) 3D Gaussians onto the 2D image plane. This is done by projecting the covariance matrix $\Sigma$ and mean $\bm{\mu}$ as $\Sigma^{\prime}=JR\Sigma R^{T}J^{T}$ and $\bm{\mu}^{\prime}=K\omega^{-1}\bm{\mu}$, where $R$ is the rotation component of world-to-camera extrinsics $\omega^{-1}$ and $J$ is the Jacobian of the affine approximation of the projective transformation []. The final pixel color $C$ and depth $D^{r}$ at pixel $\mathbf{x}^{\prime}$ is computed by blending 3D Gaussian splats that overlap at a given pixel, sorted by their depth as

$$ $C=\sum_{i\in\mathcal{N}}\mathbf{c}_{i}\alpha_{i}\prod_{j=1}^{i-1}(1-\alpha_{j} )\qquad D^{r}=\sum_{i\in\mathcal{N}}\hat{d}_{i}\alpha_{i}\prod_{j=1}^{i-1}(1- \alpha_{j})\enspace,$ (8) $$

where $\hat{d}_{i}$ is the z-axis depth of the center of the $i$-th 3D Gaussian and the final opacity $\alpha_{i}$ is the product of the opacity $o_{i}$ and the 2D Gaussian density as

$$ $\alpha_{i}=o_{i}\exp\Big{(}-\frac{1}{2}(\mathbf{x}^{\prime}-\bm{\mu}^{\prime}_ {i})^{\top}\Sigma^{\prime-1}_{i}(\mathbf{x}^{\prime}-\bm{\mu}^{\prime}_{i}) \Big{)}\enspace.$ (9) $$

Map Initialization. For every new keyframe, we adopt the RGBD strategy of MonoGS [] for adding new Gaussians to the unexplored scene space.
As we do not have access to a depth sensor, we construct a proxy depth map $D$ by combining the inlier multi-view depth $\tilde{D}$ and the monocular depth $D^{\text{mono}}$ as

$$ $D(u,v)=\begin{cases}\tilde{D}(u,v)&\text{if $\tilde{D}(u,v)$ is valid}\\ \theta D^{\text{mono}}(u,v)+\gamma&\text{otherwise}\\ \end{cases}$ (10) $$

Here, $\theta$ and $\gamma$ are computed as in eq. 3 but using depth instead of disparity.

Keyframe Selection and Optimization.
Apart from the keyframe selection based on a mean optical flow threshold $\tau$, we additionally adopt the keyframe selection strategy from [] to avoid mapping redundant frames.

To optimize the 3D Gaussian parameters, we batch the parameter updates to a local window similar to [] and apply a photometric and geometric loss to the proxy depth as well as a scale regularizer to avoid artifacts from elongated Gaussians. Inspired by [], we further use exposure compensation by optimizing an affine transformation for each keyframe. The final loss is

$$ $\mathop{\min}_{\mathcal{G},\mathbf{a},\mathbf{b}}\sum_{\begin{subarray}{c}k\in \text{KFs}\end{subarray}}\frac{\lambda}{N_{k}}\lvert(a_{k}C_{k}+b_{k})-C^{gt}_ {k}\rvert_{1}+\frac{1-\lambda}{N_{k}}\lvert D^{r}_{k}-D_{k}\rvert_{1}+\frac{ \lambda_{reg}}{|\mathcal{G}|}\sum_{i}^{|\mathcal{G}|}\lvert s_{i}-\tilde{s}_{i }\rvert_{1}\enspace,$ (11) $$

where KFs contains the set of keyframes in the local window, $N_{k}$ is the number of pixels per keyframe, $\lambda$ and $\lambda_{reg}$ are hyperparameters, $\mathbf{a}=\{a_{1},\dots,a_{k},\dots\}$ and $\mathbf{b}=\{b_{1},\dots,b_{k},\dots\}$ are the parameters for the exposure compensation and $\tilde{s}$ is the mean scaling, repeated over the three dimensions.

Map Deformation.
Since our tracking framework is globally consistent, changes in the keyframe poses and proxy depth maps need to be accounted for in the 3D Gaussian map by a non-rigid deformation. Though the Gaussian means are directly optimized, one could in theory let the optimizer deform the map as refined poses and proxy depth maps are provided. We find, however, that in particular rendering is aided by actively deforming the 3D Gaussian map. We apply the deformation to all Gaussians which receive updated poses and depths before mapping.

Each Gaussian $g_{i}$ is associated with a keyframe that anchored it to the map $\mathcal{G}$. Assume that a keyframe with camera-to-world pose $\omega$ and proxy depth $D$ is updated such that $\omega\rightarrow\omega^{\prime}$ and $D\rightarrow D^{\prime}$. We update the mean, scale and rotation of all Gaussians $g_{i}$ associated with the keyframe. Association is determined by what keyframe added the Gaussian to the scene. The mean $\bm{\mu}_{i}$ is projected into $\omega$ to find the pixel correspondence $(u,v)$. Since the Gaussians are not necessarily anchored on the surface, instead of re-anchoring the mean at $D^{\prime}$, we opt to shift the mean by $D^{\prime}(u,v)-D(u,v)$ along the optical axis. We update $R_{i}$ and $s_{i}$ accordingly as

$$ $\bm{\mu}^{\prime}_{i}=\!\bigg{(}\!1+\frac{D^{\prime}(u,v)-D(u,v)}{(\omega^{-1} \bm{\mu}_{i})_{z}}\!\bigg{)}\omega^{\prime}\omega^{-1}\bm{\mu}_{i}\enspace,R_{ i}^{\prime}=R^{\prime}R^{-1}R_{i}\enspace,s^{\prime}_{i}=\!\bigg{(}\!1+\frac{D ^{\prime}(u,v)-D(u,v)}{(\omega^{-1}\bm{\mu}_{i})_{z}}\!\bigg{)}s_{i}\enspace.$ (12) $$

Here, $(\cdot)_{z}$ denotes the z-axis depth. For Gaussians which project into pixels with missing depth or outside the viewing frustum, we only rigidly deform them. After the final global BA optimization, we additionally deform the Gaussian map and perform a set of final refinements (see suppl. material).

## 4 Experiments

We first describe our experimental setup and then evaluate our method against state-of-the-art dense RGB and RGBD SLAM methods on Replica [] as well as the real world TUM-RGBD [] and the ScanNet [] datasets.
For more experiments and details, we refer to the supplementary material.

Implementation Details.
For the proxy depth, we use $\eta=0.01$ to filter points and use the condition $n_{c}\geq 2$ to ensure multi-view consistency. For the mapping loss function, we use $\lambda=0.8$, $\lambda_{reg}=10.0$. We use 60 iterations during mapping. For tracking, we use $\alpha_{1}=0.01$ and $\alpha_{2}=0.1$ as weights for the DSPO layer. We use the flow threshold $\tau=4.0$ on ScanNet, $\tau=3.0$ on TUM-RGBD and $\tau=2.25$ on Replica. The threshold for loop detection is $\tau_{\text{loop}}=25.0$. The time interval threshold is $\tau_{t}=20$. We conducted the experiments on a cluster with an NVIDIA A100 GPU.

Evaluation Metrics.
For rendering we report PSNR, SSIM [] and LPIPS [] on the rendered keyframe images against the sensor images.
For reconstruction, we first extract the meshes with marching cubes [] as in [] and evaluate the meshes using accuracy $[cm]$, completion $[cm]$ and completion ratio $[\%]$ (threshold 5 cm) against the ground truth meshes.
We also report the re-rendering depth L1 $[cm]$ metric to the ground truth sensor depth as in [].
We use ATE RMSE $[cm]$ [] to evaluate the estimated trajectory.

Datasets.
We use the RGBD trajectories from [] captured from the synthetic Replica dataset []. We also test on real-world data using the
TUM-RGBD [] and the ScanNet [] datasets.

Baseline Methods. We compare our method to numerous published and concurrent works on dense RGB and RGBD SLAM. Concurrent works are denoted with an asterix^∗. The main baselines are GlORIE-SLAM [] and MonoGS [].

**Table 1: Rendering and Tracking Results on Replica [] for RGB-Methods. Our method outperforms all methods on rendering and performs on par for tracking accuracy. Results are from [] except ours (average over 8 scenes). Best results are highlighted as first, second, third.**
| Metric | GO-<br>SLAM [] | NICER-<br>SLAM [] | MoD-<br>SLAM^∗ [] | Photo-<br>SLAM [] | Mono-<br>GS [] | GlORIE-<br>SLAM^∗ [] | Q-SLAM<br>^∗ [] | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | 22.13 | 25.41 | 27.31 | 33.30 | 31.22 | 31.04 | 32.49 | 36.45 |
| SSIM $\uparrow$ | 0.73 | 0.83 | 0.85 | 0.93 | 0.91 | 0.91 | 0.89 | 0.95 |
| LPIPS$\downarrow$ | - | 0.19 | - | - | 0.21 | 0.12 | 0.17 | 0.06 |
| ATE RMSE<br>$\downarrow$ | 0.39 | 1.88 | 0.35 | 1.09 | 14.54 | 0.35 | - | 0.35 |

**Table 2: Rendering Performance on ScanNet []. Our method performs even better or on par with all RGB-D methods. We take the numbers for SplaTaM and Gaussian-SLAM from [].**
| Method | Metric | 0000 | 0059 | 0106 | 0169 | 0181 | 0207 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RGB-D Input |  |  |  |  |  |  |  |  |
| SplaTaM [] | PSNR$\uparrow$ | 19.33 | 19.27 | 17.73 | 21.97 | 16.76 | 19.80 | 19.14 |
| SSIM $\uparrow$ | 0.66 | 0.79 | 0.69 | 0.78 | 0.68 | 0.70 | 0.72 |  |
| LPIPS$\downarrow$ | 0.44 | 0.29 | 0.38 | 0.28 | 0.42 | 0.34 | 0.36 |  |
| \hdashline<br>MonoGS [] | PSNR$\uparrow$ | 18.70 | 20.91 | 19.84 | 22.16 | 22.01 | 18.90 | 20.42 |
| SSIM $\uparrow$ | 0.71 | 0.79 | 0.81 | 0.78 | 0.82 | 0.75 | 0.78 |  |
| LPIPS$\downarrow$ | 0.48 | 0.32 | 0.32 | 0.34 | 0.42 | 0.41 | 0.38 |  |
| \hdashline<br>Gaussian-<br>SLAM [] | PSNR$\uparrow$ | 28.54 | 26.21 | 26.26 | 28.60 | 27.79 | 28.63 | 27.67 |
| SSIM $\uparrow$ | 0.93 | 0.93 | 0.93 | 0.92 | 0.92 | 0.91 | 0.92 |  |
| LPIPS$\downarrow$ | 0.27 | 0.21 | 0.22 | 0.23 | 0.28 | 0.29 | 0.25 |  |
| RGB Input |  |  |  |  |  |  |  |  |
| GO-<br>SLAM [] | PSNR$\uparrow$ | 15.74 | 13.15 | 14.58 | 14.49 | 15.72 | 15.37 | 14.84 |
| SSIM $\uparrow$ | 0.42 | 0.32 | 0.46 | 0.42 | 0.53 | 0.39 | 0.42 |  |
| LPIPS$\downarrow$ | 0.61 | 0.60 | 0.59 | 0.57 | 0.62 | 0.60 | 0.60 |  |
| MonoGS [] | PSNR$\uparrow$ | 16.91 | 19.15 | 18.57 | 20.21 | 19.51 | 18.37 | 18.79 |
| SSIM $\uparrow$ | 0.62 | 0.69 | 0.74 | 0.74 | 0.75 | 0.70 | 0.71 |  |
| LPIPS$\downarrow$ | 0.70 | 0.51 | 0.55 | 0.54 | 0.63 | 0.58 | 0.59 |  |
| \hdashline<br>GlORIE-<br>SLAM^∗ [] | PSNR$\uparrow$ | 23.42 | 20.66 | 20.41 | 25.23 | 21.28 | 23.68 | 22.45 |
| SSIM $\uparrow$ | 0.87 | 0.87 | 0.83 | 0.84 | 0.91 | 0.76 | 0.85 |  |
| LPIPS$\downarrow$ | 0.26 | 0.31 | 0.31 | 0.21 | 0.44 | 0.29 | 0.30 |  |
| \hdashline<br>Splat-SLAM<br>(Ours) | PSNR$\uparrow$ | 28.68 | 27.69 | 27.70 | 31.14 | 31.15 | 30.49 | 29.48 |
| SSIM $\uparrow$ | 0.83 | 0.87 | 0.86 | 0.87 | 0.84 | 0.84 | 0.85 |  |
| LPIPS $\downarrow$ | 0.19 | 0.15 | 0.18 | 0.15 | 0.23 | 0.19 | 0.18 |  |

## Appendix A Method

We describe further details about our method that were left out from the main paper.

Comparison to Existing Works. To further clarify the differences between our method and existing 3DGS SLAM works, we classify each method in tab. S8 based on important characteristics.
It shows that our work is the first to include loop closure, proxy depth, RGB-only and online 3D Gaussian deformations.

**Table S8: Method Classification. We show that our method is the first to combine 3D Gaussian SLAM with loop closure, proxy depth and online 3D Gaussian map deformations in an RGB-only SLAM system.**
|  | RGB-only | Loop<br>Closure | Proxy<br>Depth | Online 3DGS<br>Deformations |
| --- | --- | --- | --- | --- |
| GS-SLAM [] | ✗ | ✗ | ✓ | ✗ |
| Gaussian-SLAM [] | ✗ | ✗ | ✓ | ✗ |
| SplaTaM [] | ✗ | ✗ | ✓ | ✗ |
| MonoGS [] | ✓ | ✗ | ✗ | ✗ |
| Photo-SLAM [] | ✓ | ✓ | ✗ | ✗ |
| Splat-SLAM (ours) | ✓ | ✓ | ✓ | ✓ |

Map Initialization.
With map initialization, we refer to the process of anchoring new Gaussians during scene exploration. For every new keyframe to be mapped, we adopt the strategy that MonoGS [] uses in pure RGBD mode. It works by unprojecting the depth reading per pixel to 3D and then downsampling this point cloud by a factor $\theta$. New Gaussians are then assigned their means as the point cloud. The rotations are initialized to identity, the opacity to 0.5 and the scales are initialized related to their distance to the nearest neighbor point in the point cloud.

Keyframe Selection and Local Windowing.
As mentioned in the main paper, we adopt the keyframe selection strategy from MonoGS []. We describe this strategy in the following.

Keyframes are selected based on the covisibility of the Gaussians. Between two keyframes $i$ and $j$, the covisibility is defined using the Intersection over Union (IOU) and Overlap Coefficient (OC):

$$ $\text{IOU}_{\text{cov}}(i,j)=\frac{|\mathcal{G}_{v}^{i}\cap\mathcal{G}_{v}^{j} |}{|\mathcal{G}_{v}^{i}\cup\mathcal{G}_{v}^{j}|},$ (16) $$

$$ $\text{OC}_{\text{cov}}(i,j)=\frac{|\mathcal{G}_{v}^{i}\cap\mathcal{G}_{v}^{j}| }{\min(|\mathcal{G}_{v}^{i}|,|\mathcal{G}_{v}^{j}|)},$ (17) $$

where $\mathcal{G}_{v}^{i}$ are the Gaussians visible in keyframe $i$, based on the following definition of visibility. A Gaussian is seen as visible from a camera pose if it is used in the rasterization pipeline when rendering and if the accumulated transmittance $\prod_{j=1}^{i-1}(1-\alpha_{j})$ has not yet reached 0.5.

A keyframe $i$ is added to the keyframe window KFs if, given the last keyframe $j$, $\text{IOU}_{\text{cov}}(i,j)<k_{\text{fcov}}$ or if the relative translation $t_{ij}>k_{\text{fm}}\hat{D}_{i}$, where $\hat{D}_{i}$ is the median depth of frame $i$. For Replica, $k_{\text{fcov}}=0.95$, $k_{\text{fm}}=0.04$ and for TUM and ScanNet, $k_{\text{fcov}}=0.90$, $k_{\text{fm}}=0.08$. The registered keyframe $j$ in KFs is removed if $\text{OC}_{\text{cov}}(i,j)<k_{\text{fc}}$, where keyframe $i$ is the latest added keyframe. For all datasets, the cutoff is set to $k_{\text{fc}}=0.3$. The size of the keyframe window is set to $|\text{KFs}|=10$ for Replica and $|\text{KFs}|=8$ for TUM and ScanNet.

Pruning and Densification
We also follow [] when it comes to Gaussian pruning and densification. Pruning is done based on the visibility: if new Gaussians inserted
within the last 3 keyframes are not visible by at least 3
other frames in the keyframe window KFs, they are removed. Visibility-based pruning is only done when the keyframe window KFs is full. Additionally, every 150 mapping iterations, Gaussians with opacity lower than 0.7 are removed globally. Also Gaussians which project in 2D with a too large scale are removed. Densification is done as in [], also at an interval of every 150 mapping iterations.

Final Refinement.
Similar to GlORIE-SLAM [], which performs a final refinement after the last final global BA at the end of the trajectory, we also perform a few refinement iterations after the last final global BA. Also MonoGS [] performs a set of final iterations at the end of the SLAM trajectory to refine the colors.

Our refinement strategy is straight forward. We disable pruning and densification of the Gaussians and perform a set of optimization iterations $\beta$ using the same loss function as in the main paper, but only sampling random single frames per iteration.

Differences to GlORIE-SLAM.
We briefly discuss some differences to GlORIE-SLAM [] not covered in the main paper. GlORIE-SLAM uses an additional point cloud called $P_{d}$ consisting of all inlier mullti-view depth maps unprojected into a point cloud. We found that this is not needed and it saves memory and compute to not use it. GlORIE-SLAM also re-anchors the neural points at the depth reading. We do not do this as the Gaussians do not necessarily lie on the surface exactly. Finally, GlORIE-SLAM requires input depth to guide the sampling of points to render color and depth. If the depth is noisy or if the map is used for tracking (*i.e.* frame-to-model tracking), the depth guiding strategy is not favorable as it leads to artifacts when sampling the wrong points (when noisy depth is encountered) and to a much smaller basin of convergence when tracking (because the rendering is conditioned on the current view point). With 3D Gaussians, we can avoid depth guidance during rendering.

## Appendix B More Experiments

To accompany the evaluations provided in the main paper, we provide further experiments in this section.

Implementation Details.
As the point cloud downsampling factor, we use $\theta=32$ for all frames but the first frame where $\theta=16$ is used. We use $\beta=2000$, the number of iterations for the final refinement optimization, on the Replica dataset and $\beta=26000$ on the TUM-RGBD [] and ScanNet [] datasets (same as MonoGS []). We benchmark the runtime on an AMD Ryzen Threadripper Pro 3945WX 12-Cores with an NVIDIA GeForce RTX 3090 Ti with 24 GB of memory. For the remaining hyperparameters, we refer to MonoGS [] for the Gaussian mapping and GlORIE-SLAM for tracking [].

A Note on Rendering and Runtime with MonoGS. By default, MonoGS [] does not evaluate the rendering error on the mapped keyframes nor implement the exposure compensation during rendering evaluation. To compare our results fairly to MonoGS, we implement these details and run the experiments with these settings enabled. Further, we report the runtime for MonoGS using a single process (same as us) compared to the reported number in the paper, which was using multiple processes at once.

A Note on Gaussian Deformation with Photo-SLAM. Though not fully clear from reading the paper, after discussing with the authors of Photo-SLAM [], we find that they do, in fact, not deform the Gaussians as a result of global BA or loop closure. They found this to be unstable in their experiments. This suggests that our deformation strategy is non-trivial.

Justification of Monocular Depth Estimator.
There are already numerous monocular depth estimators, but most of them are limited by speed, memory or quality. We use Omnidata [] since empirically we found it still provides the best trade-off between output performance and runtime. We also tested our system with Depth Anything [], but found that it was marginally worse in terms of the final reconstructed mesh accuracy.

### B.1 Tracking on ScanNet and TUM-RGBD

We do not put the results on tracking for ScanNet and TUM-RGBD since we use the tracking framework from GlORIE-SLAM [], but we provide the numbers here. Tab. S9 and tab. S10 show the tracking accuracy of the estimated trajectory on ScanNet [] and TUM-RGBD [] respectively. Our method shows competitive results in every single scene and gives the best average value among the RGB and RGB-D methods.

**Table S9: Tracking Accuracy ATE RMSE [cm] $\downarrow$ on ScanNet []. Our method equals to GlORIE-SLAM [], giving the average lowest trajectory error. Results for the RGB-D methods are from []. Note that all methods with a ^∗ are concurrent works.**
| Method | 0000 | 0059 | 0106 | 0169 | 0181 | 0207 | Avg.-6 | 0054 | 0233 | Avg.-8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RGB-D Input |  |  |  |  |  |  |  |  |  |  |
| NICE-SLAM [] | 12.0 | 14.0 | 7.9 | 10.9 | 13.4 | 6.2 | 10.7 | 20.9 | 9.0 | 11.8 |
| Co-SLAM [] | 7.1 | 11.1 | 9.4 | 5.9 | 11.8 | 7.1 | 8.7 | - | - | - |
| ESLAM [] | 7.3 | 8.5 | 7.5 | 6.5 | 9.0 | 5.7 | 7.4 | 36.3 | 4.3 | 10.6 |
| MonoGS[] | 16.1 | 6.4 | 8.1 | 8.7 | 26.4 | 9.2 | 12.5 | 20.6 | 13.1 | 13.6 |
| RGB Input |  |  |  |  |  |  |  |  |  |  |
| MonoGS[] | 149.2 | 96.8 | 155.5 | 140.3 | 92.6 | 101.9 | 122.7 | 206.4 | 89.1 | 129.0 |
| GO-SLAM [] | 5.9 | 8.3 | 8.1 | 8.4 | 8.3 | 6.9 | 7.7 | 13.3 | 5.3 | 8.1 |
| HI-SLAM[] | 6.4 | 7.2 | 6.5 | 8.5 | 7.6 | 8.4 | 7.4 | - | - | - |
| Q-SLAM^∗[] | 5.8 | 8.5 | 8.4 | 8.7 | 8.8 | - | - | 12.6 | 5.3 | - |
| GlORIE-SLAM^∗ [] | 5.5 | 9.1 | 7.0 | 8.2 | 8.3 | 7.5 | 7.6 | 9.4 | 5.1 | 7.5 |
| Ours | 5.5 | 9.1 | 7.0 | 8.2 | 8.3 | 7.5 | 7.6 | 9.4 | 5.1 | 7.5 |

**Table S10: Tracking Accuracy ATE RMSE [cm] $\downarrow$ on TUM-RGBD []. Our method equals to GlORIE-SLAM [], giving the average lowest trajectory error. Note that all methods with a ^∗ are concurrent works.**
| Method | f1/desk | f2/xyz | f3/off | Avg.-3 | f1/desk2 | f1/room | Avg.-5 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| RGB-D Input |  |  |  |  |  |  |  |
| SplaTAM [] | 3.4 | 1.2 | 5.2 | 3.3 | 6.5 | 11.1 | 5.5 |
| GS-SLAM^∗ [] | 1.5 | 1.6 | 1.7 | 1.6 | - | - | - |
| GO-SLAM [] | 1.5 | 0.6 | 1.3 | 1.1 | - | 4.7 | - |
| MonoGS [] | 1.4 | 1.4 | 1.5 | 1.5 | 5.1 | 6.3 | 3.1 |
| RGB Input |  |  |  |  |  |  |  |
| MonoGS [] | 3.8 | 5.2 | 2.9 | 4.0 | 75.7 | 76.6 | 32.8 |
| Photo-SLAM [] | 1.5 | 1.0 | 1.3 | 1.3 | - | - | - |
| DIM-SLAM [] | 2.0 | 0.6 | 2.3 | 1.6 | - | - | - |
| GO-SLAM [] | 1.6 | 0.6 | 1.5 | 1.2 | 2.8 | 5.2 | 2.3 |
| MoD-SLAM^∗ [] | 1.5 | 0.7 | 1.1 | 1.1 | - | - | - |
| Q-SLAM^∗ [] | 1.3 | 0.9 | - | - | 2.3 | 4.9 | - |
| GlORIE-SLAM^∗ [] | 1.6 | 0.2 | 1.4 | 1.1 | 2.8 | 4.2 | 2.1 |
| Ours | 1.6 | 0.2 | 1.4 | 1.1 | 2.8 | 4.2 | 2.1 |

### B.2 Full Evaluations Data

**Table S11: Full Evaluation on Replica []. We show the ATE RMSE [cm] evaluation on the keyframes as well as on the full trajectory.**
|  |  | Metric | R-0 | R-1 | R-2 | O-0 | O-1 | O-2 | O-3 | O-4 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Reconstruction | Render Depth L1 $\downarrow$ | 2.90 | 2.16 | 2.18 | 2.44 | 1.97 | 2.46 | 2.62 | 2.53 | 2.41 |  |
| Accuracy $\downarrow$ | 1.99 | 1.91 | 2.06 | 3.96 | 2.03 | 3.45 | 2.15 | 1.89 | 2.43 |  |  |
| Completion $\downarrow$ | 3.78 | 3.38 | 3.34 | 2.75 | 3.33 | 4.36 | 3.96 | 4.25 | 3.64 |  |  |
| Comp. Rat. $\uparrow$ | 85.47 | 86.88 | 86.12 | 87.32 | 85.17 | 81.37 | 82.25 | 82.95 | 84.69 |  |  |
| Rendering | Keyframes | PSNR $\uparrow$ | 32.25 | 34.31 | 35.95 | 40.81 | 40.64 | 35.19 | 35.03 | 37.40 | 36.45 |
| SSIM $\uparrow$ | 0.91 | 0.93 | 0.95 | 0.98 | 0.97 | 0.96 | 0.95 | 0.98 | 0.95 |  |  |
| LPIPS $\downarrow$ | 0.10 | 0.09 | 0.06 | 0.05 | 0.05 | 0.07 | 0.06 | 0.04 | 0.06 |  |  |
| Tracking | Keyframes<br>Trajectory | ATE<br>RMSE<br>$\downarrow$ | 0.29 | 0.38 | 0.24 | 0.27 | 0.35 | 0.34 | 0.42 | 0.43 | 0.34 |
| \cdashline2-12 | Full<br>Trajectory | ATE<br>RMSE<br>$\downarrow$ | 0.29 | 0.33 | 0.25 | 0.29 | 0.35 | 0.34 | 0.42 | 0.43 | 0.34 |
| Number of<br>Gaussians | 1000x |  | 116 | 116 | 91 | 76 | 66 | 134 | 114 | 106 | 102 |

### B.3 Influence of Monocular Depth

**Table S14: Full Evaluations on Replica [] with ground truth depth. Both reconstruction and rendering results improve significantly with the ground truth depth, suggesting that our method is bounded by the quality of current day monocular depth estimation. Since we do not require any extra training or fine-tuning of the monocular depth estimator, it is easy to plug in a better estimator once available. Tracking performance does not change much.**
|  |  | Metric | R-0 | R-1 | R-2 | O-0 | O-1 | O-2 | O-3 | O-4 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Recon-<br>struction | Render Depth L1 $\downarrow$ | 2.38 | 1.31 | 1.73 | 1.15 | 1.60 | 1.29 | 5.71 | 1.93 | 2.14 |  |
| Accuracy $\downarrow$ | 1.29 | 0.91 | 1.05 | 1.22 | 0.83 | 0.96 | 1.24 | 1.07 | 1.07 |  |  |
| Completion $\downarrow$ | 3.43 | 2.83 | 2.66 | 1.50 | 2.46 | 3.57 | 3.46 | 3.61 | 2.94 |  |  |
| Comp. Rat. $\uparrow$ | 86.61 | 88.69 | 88.70 | 93.44 | 89.09 | 85.20 | 84.60 | 85.32 | 87.71 |  |  |
| Rendering | PSNR $\uparrow$ | 35.66 | 37.65 | 38.87 | 43.95 | 43.28 | 37.93 | 37.41 | 39.88 | 39.33 |  |
| SSIM $\uparrow$ | 0.96 | 0.96 | 0.97 | 0.99 | 0.98 | 0.96 | 0.96 | 0.98 | 0.97 |  |  |
| LPIPS $\downarrow$ | 0.04 | 0.05 | 0.03 | 0.02 | 0.02 | 0.06 | 0.04 | 0.03 | 0.04 |  |  |
| Tracking |  | ATE<br>RMSE<br>$\downarrow$ | 0.29 | 0.38 | 0.24 | 0.28 | 0.39 | 0.35 | 0.45 | 0.40 | 0.35 |

While we show that the monocular depth improves the geometric estimation capability of our framework, it may still be erroneous. To better understand the accuracy of the monocular depth, we replace it with the ground truth sensor depth instead. This experiment acts as the upper bound of our method if the monocular depth is perfect. The experiments are done on Replica [] and are shown in tab. S14. Compared with the standard setting with the monocular depth, the ground truth depth setting gives improvements on both reconstruction and rendering quality, which reveals that our method still has potential to achieve better mapping results once better monocular depth is available. Since our method does not require further training or fine-tuning for the monocular depth, it is quite easy to just replace the current off-the-shelf monocular depth estimator with a better one.

### B.4 Impact of Deformation

During runtime, we deform the 3D Gaussian map to account for adjustments to poses and depth that have already been integrated into the existing map. An alternative to performing the deformation is to solely rely on optimization to resolve the new map. We conduct two experiments to show the benefit of performing the deformation, especially when it comes to rendering accuracy. In tab. S15, we vary the number of final refinement iterations and evaluate the rendering depth L1 and PSNR on the Replica office 0 scene. We find that utilizing online 3D Gaussian deformations yields better rendering and depth L1 accuracy regardless of the number of iterations. In tab. S16 we conduct the same experiment, but over a set of scenes on ScanNet. We find that on average, by enabling the deformation, we achieve higher rendering accuracy and lower depth L1 error. The improvement is, however, more significant when it comes to the rendering accuracy.

**Table S15: Gaussian Deformation Ablation on Replica [] office 0.**
| Nbr of Final Iterations $\beta$ |  | Metric | 0K | 0.5K | 1K | 2K |
| --- | --- | --- | --- | --- | --- | --- |
| Reconstruction | W/O Deform<br>W Deform | Render Depth L1 $\downarrow$ | 8.84 | 3.49 | 2.64 | 2.6 |
| Render Depth L1 $\downarrow$ | 6.55 | 2.37 | 2.34 | 2.40 |  |  |
| Rendering | W/O Deform<br>W Deform | PSNR $\uparrow$ | 22.86 | 34.30 | 37.66 | 37.86 |
| PSNR $\uparrow$ | 30.50 | 39.87 | 40.59 | 41.20 |  |  |

**Table S16: Gaussian Deformation Ablation on ScanNet [].**
|  |  | Metric | 0000 | 0054 | 0059 | 0106 | 0169 | 0181 | 0207 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Rendering | W/O Deform<br>W Deform | PSNR$\uparrow$ | 25.15 | 28.39 | 27.77 | 25.25 | 29.41 | 30.38 | 29.30 | 27.95 |
| PSNR$\uparrow$ | 28.68 | 30.21 | 27.69 | 27.70 | 31.14 | 31.15 | 30.49 | 29.58 |  |  |
| Depth<br>Rendering | W/O Deform<br>W Deform | L1$\downarrow$ [cm] | 7.86 | 22.81 | 10.51 | 24.19 | 11.54 | 18.48 | 13.66 | 15.58 |
| 8.24 | 18.24 | 13.39 | 23.5 | 11.49 | 18.35 | 13.78 | 15.28 |  |  |  |

### B.5 Final Refinement Iterations

After the final global BA step, we perform a final refinement, similar to MonoGS[], but include the geometric depth loss as well and do not only refine with a color loss. We ablate the influence on the results by varying the number of iterations of the final refinement in tab. S17. We find that the rendering accuracy increases monotonically with the number of iterations while the geometric accuracy decreases with more than 2K iterations. We believe this to be a result of fitting to the noisy monocular depth. We choose to use 2K iterations since this provides the best trade-off between rendering and geometric accuracy. 2K iterations takes around 15 seconds on our benchmark hardware which consists of an AMD Ryzen Threadripper Pro 3945WX 12-Cores with an NVIDIA GeForce RTX 3090 Ti with 24 GB of memory.

**Table S17: Final Refinement Iterations Ablation on Replica []. The results are averaged over the 8 scenes.**
| Nbr of Final Iterations $\beta$ |  | Metric | 2K | 5K | 10K | 26K |
| --- | --- | --- | --- | --- | --- | --- |
| Reconstruction | Render Depth L1 $\downarrow$ | 2.36 | 2.45 | 2.51 | 2.59 |  |
| Accuracy $\downarrow$ | 2.46 | 2.66 | 2.84 | 3.02 |  |  |
| Completion $\downarrow$ | 3.60 | 3.61 | 3.59 | 3.60 |  |  |
| Comp. Rat. $\uparrow$ | 84.87 | 84.71 | 84.80 | 84.77 |  |  |
| Rendering | Keyframes | PSNR $\uparrow$ | 36.77 | 37.80 | 38.41 | 38.95 |

### B.6 Impact of Downsampling Factor

During mapping, the point cloud formed from unprojecting the depth input is downsampled to avoid adding redundant Gaussians to the scene representation. We investigate the impact of using stronger versus weaker downsampling in tab. S18 where we also compare to the sensitivity of MonoGS[] with respect to the same parameter. Tab. S18 shows that both systems are not very sensitive to the model compression as a result of a larger downsampling factor $\theta$. When both systems use the same number of Gaussians on average ($\theta=32$ for MonoGS and $\theta=64$ for our method), we find that our method performs significantly better in terms of depth rerendering and photometric accuracy. For all results in the main paper, we use $\theta=32$.

**Table S18: Downsampling Factor $\theta$ Ablation on Replica []. The results are averaged over the 8 scenes.**
| Downsampling Factor $\theta$ |  | Metric | 16 | 32 | 64 |
| --- | --- | --- | --- | --- | --- |
| Reconstruction | Ours<br>MonoGS [] | Render Depth L1 $\downarrow$ | 2.38 | 2.40 | 2.46 |
| 33.43 | 28.47 | 28.09 |  |  |  |
| Rendering | Ours<br>MonoGS [] | PSNR $\uparrow$ | 36.63 | 36.45 | 36.31 |
| 31.17 | 30.87 | 29.64 |  |  |  |
| Number of<br>Gaussians | Ours<br>MonoGS [] | 1000x$\downarrow$ | 141 | 102 | 83 |
| 97 | 83 | 73 |  |  |  |

### B.7 Runtime Evaluation

To be consistent with the keyframe selection hyperparameters of MonoGS [], we report on the same parameters as MonoGS uses by default. In practice, this means that few keyframes from the tracking system (determined via mean optical flow thresholding) are actually filtered out and not mapped. In tab. S19, we show that by altering the hyperparamters, we can speed up the system during runtime, while still rendering and reconstructing the scene well. Note that we evaluate the rendering performance on the same set of views for all runs. We benchmark the runtime on an AMD Ryzen Threadripper Pro 3945WX 12-Cores with an NVIDIA GeForce RTX 3090 Ti with 24 GB of memory. We note that we currently do not leverage multiprocessing to the amount possible in practice *i.e.* currently we first do tracking and then mapping *i.e.* there is no simultaneous tracking and mapping. This is, however, straightforward to include, which should further speed up the runtime.

**Table S19: Keyframe Hyperparameter Search on Replica [] room 0. By changing the keyframe selection hyperparameters, we can speed up our runtime without impacting reconstruction and rendering too much. We evaluate the rendering performance on the same set of frames for all runs. In comparison, with the default $k_{\text{fcov}}=0.95$, $k_{\text{fm}}=0.04$, MonoGS [] yields PSNR: 26.12 and render depth L1: 17.38 cm.**
| $k_{\text{fcov}}$, $k_{\text{fm}}$ |  | 0.95, 0.04 | 0.90, 0.08 | 0.85, 0.08 | 0.80, 0.12 | 0.70, 0.16 | 0.60, 0.20 | 0.50, 0.30 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Reconstruction | Render Depth L1 $\downarrow$ | 2.90 | 2.94 | 2.97 | 3.08 | 3.37 | 3.53 | 4.78 |
| Accuracy $\downarrow$ | 1.99 | 1.94 | 2.06 | 2.04 | 2.54 | 3.20 | 6.20 |  |
| Completion $\downarrow$ | 3.78 | 3.76 | 3.79 | 3.77 | 3.86 | 3.93 | 5.23 |  |
| Comp. Rat. $\uparrow$ | 85.47 | 85.58 | 85.39 | 85.53 | 85.03 | 84.33 | 80.38 |  |
| Rendering | PSNR $\uparrow$ | 32.25 | 31.65 | 31.31 | 30.59 | 30.12 | 29.25 | 27.59 |
| Runtime | FPS $\uparrow$ | 1.24 | 1.45 | 1.62 | 2.02 | 2.50 | 3.03 | 3.67 |

### B.8 Additional Qualitative Reconstructions

In fig. S6 we show additional qualitative results from the Replica dataset on normal shaded meshes.

Figure: Figure S6: Reconstruction Results on Replica []. Our method improves upon the geometric accuracy compared to existing works, when observing the normal shaded meshes. In particular, GlORIE-SLAM suffers from floating point artifacts. MonoGS suffers badly from a lack of proxy depth, despite multiview optimization.
Refer to caption: extracted/5621324/figs/reconstruction/glorie_off4.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: How it performs on dataset that contains more forward camera motion such as TartanAir?
While 3DGS itself might perform poorly or fail, it is worth to study the camera pose itself and depth map.
soundness: 3
strengths: 1. The implementation of the system is sophisticated which requires extensive effort.
2. The utilization of monocular depth offers good regularization of depth map while the depth are still calculated per-pixel which maintains the accuracy.
3. The performance, especially the global consistency is superior to other global consistent 3DGS baselines.
summary: The paper introduces a two-stage system to reconstruct globally consistent 3DGS. In the first stage, the system reconstructs camera poses and metric depth at each keyframe through dense bundle adjustment, where monocular depth estimation is leveraged for proxy depth fusion and additional constraints for dense BA. In the second stage, a globally consistent 3DGS is reconstructed in an incremental fashion where the 3DGS parameters will be updated once a new keyframe is added. To validate the effectiveness of the proposed method, the authors conducted experiments on existing public benchmarks and demonstrated better efficiency and performance over existing baselines in globally consistent 3DGS reconstruction.
weaknesses: 1. The paper is more about a sophisticated system implementation for a specific problem and lacks some insights or contributions to the understanding of the problem. Thus, it is better to be published at other venues such as CVPR and 3DV.
2. The term "Deformable" is a little bit confusing since it is widely referred to deformable structure in computer vision literatures. Thus, it is better to replace this term with other word.
3. There are some missing but important details in Figure.2:
    - How the conv-gru facilitates the loop closure?
    - What is the flow revision? I believe it is refinement based on dense optical flow but it is better to be clearly indicated.
    - What is the \hat{D}, it should be the output of filtered depth, but should be included in one of the equations.
4. The DSPO is refereed as a "layer". However, it is only used in forward optimization in the paper and there is no backward backpropgation through it.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: - How does the deformed map relate to RGB-only SLAM? Given that it utilizes depth information, this approach appears equally suitable for RGB-D SLAM. Addressing BA-induced map inconsistencies, as tackled by previous works like Loopy-SLAM and LoopSplat, is not unique to monocular systems. Therefore, the rationale for introducing a deformed map specifically within monocular SLAM is not convincingly articulated.

- Compared to LoopySLAM, which is based on NeRF, and LoopSplat, which uses Gaussian splatting, the primary distinction appears to be whether map deformation is applied per keyframe or per submap. Given the efficiency perspective, adjusting the submap seems more practical and feasible, as adjustments between adjacent keyframes are typically minimal. To validate this approach, further experiments focusing on efficiency and rendering quality are recommended.

- The paper employs metrics like PSNR, SSIM, and LPIPS to assess rendering quality. It remains unclear how these metrics are configured for evaluation—are they assessed at each keyframe during the SLAM system's runtime, post-mapping process, or on the reconstructed mesh? Further clarification on the evaluation setup for these metrics would enhance the understanding of their application and relevance.

- A typo on line 24,  RGB-only SLAM methods methods.
soundness: 2
strengths: - The paper conducted experiments on both synthetic and real-world datasets for tracking and mapping evaluation
- The paper writing is comprehensive and easy to understand
- The paper presents high performance among the baseline methods
summary: The paper proposed an RGB-only SLAM system that utilizes a deformed 3D Gaussian representation for mapping, and a raft-based monocular tracking system with additional depth estimation. The system was evaluated on Replica, TUM, and ScanNet datasets, comparing with both NeRF-based and Gaussian-based SLAM systems, achieving superior or on-par performance with existing methods in tracking,
mapping and rendering accuracy while yielding small map sizes and fast runtimes.
weaknesses: - Lack of novelty. The paper significantly derives its methods from Droid-SLAM, especially in monocular tracking based on raft, with the addition of a depth estimation module previously introduced in Glorie-SLAM, for which appropriate credit is lacking in the methods section. Moreover, although the proposed deformed map offers a partial solution to BA-induced inconsistencies, it heavily relies on the existing monocular tracking method, which has limited contribution to addressing the core challenges of RGB-only SLAM (e.g. depth and scale estimation). Finally, the idea of loop-induced scene drift is not new, similar ideas have been proposed from LoopySLAM, LoopSplat etc.

- If the system employs a pre-existing tracking module for pose estimation and uses a Gaussian map for scene representation, it is essential to compare it with similar frameworks to highlight distinct advantages or improvements. For instance, Photo-SLAM (as mentioned in supplementary material) integrates ORB-SLAM3 for tracking, while RTG-SLAM incorporates ORB-SLAM2, both supporting monocular mode and utilizing 3DGS for map representation. This comparison is crucial for evaluating the proposed system within the context of existing solutions.

- Although the paper demonstrates high performance, the motivation for each proposed component is either derived from existing works or lacks clear connections. Given the focus of the ICLR conference on presenting novel insights to the community, this paper may not fully align with the conference's objectives.

References:
- Teed, Zachary, and Jia Deng. "Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras." Advances in neural information processing systems 34 (2021): 16558-16569.
- Zhang, Ganlin, et al. "Glorie-slam: Globally optimized rgb-only implicit encoding point cloud slam." arXiv preprint arXiv:2403.19549 (2024).
- Liso, Lorenzo, et al. "Loopy-slam: Dense neural slam with loop closures." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
- Zhu, Liyuan, et al. "Loopsplat: Loop closure by registering 3d gaussian splats." arXiv preprint arXiv:2408.10154 (2024).
- Huang, Huajian, et al. "Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular Stereo and RGB-D Cameras." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
- Peng, Zhexi, et al. "Rtg-slam: Real-time 3d reconstruction at scale using gaussian splatting." ACM SIGGRAPH 2024 Conference Papers. 2024.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1)  Could you further explain how to construct proxy depth better in detail?
soundness: 3
strengths: 1）	This paper is the first RGB-only 3DGS-based SLAM system with loop closure, proxy depth, and online 3D Gaussian map deformations with improved map sizes and runtimes. In my opinion, this is a relatively comprehensive work in the field of 3DGS-based SLAM. As loop closure is a crucial challenge in SLAM, this work enables map deformations at loop closure and integrates global bundle adjustment.

2）	Extensive evaluations across multiple datasets demonstrate accurate tracking and high-quality rendering. The paper also shows the method's advantages in terms of runtime and storage.

3）	Figure 3 and Figure 4 effectively show the advantages of Splat-SLAM over other methods in rendering and reconstruction.

4）	The paper is well-written, clearly explaining the design rationale behind each component. For instance, the authors justify the choice of the DPT model for proxy depth estimation over newer alternatives.
summary: This paper presents an RGB-only 3DGS-based SLAM system addressing key limitations in existing 3DGS-based SLAM approaches, e.g.,  challenges in achieving global map optimization and accurate 3D reconstructions. The proposed method, Splat-SLAM, supports map deformations upon loop closure and incorporates global bundle adjustment (BA) while leveraging a monocular depth estimation model to enhance reconstruction accuracy.
weaknesses: 1)  It would be better to include visualizations that illustrate loop closure results. For example, presenting complete reconstruction results of selected scenes and comparing these results across your method, ground truth, and other approaches would offer valuable insights.

2)  Minor: 
Wrong numbers are highlighted in Table 1.  The section on Influence of Monocular Depth in the supplementary material is interesting, as it illustrates the upper bound of your approach. Since this content enhances the main paper, it may be worth considering moving this section to the main text.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. According to Table 6, Splat-SLAM achieves superior memory usage and runtime. I noticed the running time in Tab.6, can author provide a time breakdown of each component? How many 3dgs point does the model use?

2. Table 6 shows that SPLAT-SLAM requires a mere 6.5MB of storage, which is equivalent to a substantial 135,416 Gaussian points (6.5 / 4 / (3 + 3 + 4 + 1 + 1) * 10^6) = 135416 Gaussians, is nothing short of remarkable. Unfortunately, the lack of a demonstration video to substantiate this claim is a significant omission.

3. In Section 3.2, the authors introduce a deformable 3D Gaussian scene representation[4], which may be perceived as misleading. Deformable Gaussians are conventionally employed in 4D reconstruction, whereas this section elaborates on the optimization process of 3DGS maps subject to loss function constraints via DSPO or loop closure optimizations. The terminology 'deformable 3D Gaussian' is not precisely applicable in this context. I would appreciate it if the authors could provide clarification on this matter.

I have assigned a borderline rating to this manuscript. However, I am more than willing to engage in a constructive discussion and revisit my evaluation in light of the authors' responses to my comments
soundness: 3
strengths: The proposed SLAM system combines the strengths of frame-to-frame tracking using recurrent dense optical flow with the fidelity of 3D Gaussians as the map representation without the dependence of depth inputs.
The proposed SLAM system performs better than existing RGB-only SLAM methods in tracking, mapping, and rendering accuracy, and more importantly, yielding small map sizes and fast runtimes.
The paper is well-written, concise, and has excellent formatting of figures and formulas.
summary: This paper addresses the challenges of existing methods of worse reconstruction quality due to the lacing of the global map and pose optimization or monocular depth utilization. The author proposes a dense RGB-only SLAM system with 3D Gaussian representation and utilizes global optimized tracking by actively deforming the 3D Gaussian map. Experiments on the Replica, TUM-RGBD, and ScanNet datasets demonstrate the effectiveness of the proposed RGB-only SLAM methods.
weaknesses: 1. The author suggests that this paper proposed the first RGB-only SLAM system with a dense 3D Gaussian map that globally optimized tracking by adapting dynamically to keyframe pose and depth updates. Nevertheless, I argue that this claim is not entirely accurate, since certain prior studies, notably PhotoSLAM, support monocular video as well.

2. Notably, the proposed method falls short of achieving real-time performance, with a frame rate of only 1.24 FPS on the simplest Replica room0 dataset (Table 6). This limitation will hinder the applicability of this method to large-scale, real-world environments. This might related to the depth estimation module for proxy depth prediction.

3. Upon reviewing the paper, I believe that it fails to demonstrate sufficient innovation in the context of 3DGS scene representation, as many of the techniques, e.g., loop closure, global BA., and proxy depth map optimization, have been previously explored in existing literature[1][2][3].

4. Despite the method's claim of not requiring depth input, the algorithm's underlying architecture still employs a depth estimator to compute depth, which are subsequently used to calculate the depth rendering loss and inform pose and map optimization. As a result, the proposed method still fundamentally relies on depth estimation, which in turn contributes to slower online SLAM algorithm performance.

5. Although the paper's experimental design is thorough, it still exhibits some limitations. Primarily, for SLAM algorithms, the accuracy of tracking and the quality of the generated map are of paramount importance. Consequently, rendering quality is somewhat secondary. The authors should prioritize including more figures illustrating tracking accuracy in the main text, rather than relegating them to the supplementary materials. Additionally, I highly recommend that the authors incorporate a comparison with PhotoSLAM to further bolster the experimental setup.

[1] Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping
for Monocular, Stereo, and RGB-D Cameras
[2] plaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM
[3] NICER-SLAM Neural Implicit Scene Encoding for RGB SLAM
[4] Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction