DG-SLAM: Robust Dynamic Gaussian Splatting SLAM with Hybrid Pose Optimization

Authors: yueming  xu
haochen  jiang
zhongyang  xiao
jianfeng  feng
li  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 14:31:22

Abstract: Achieving robust and precise pose estimation in dynamic scenes is a significant research challenge in Visual Simultaneous Localization and Mapping (SLAM). Recent advancements integrating Gaussian Splatting into SLAM systems have proven effective in creating high-quality renderings using explicit 3D Gaussian models, significantly improving environmental reconstruction fidelity. However, these approaches depend on a static environment assumption and face challenges in dynamic environments due to inconsistent observations of geometry and photometry. To address this problem, we propose DG-SLAM, the first robust dynamic visual SLAM system grounded in 3D Gaussians, which provides precise camera pose estimation alongside high-fidelity reconstructions. Specifically, we propose effective strategies, including motion mask generation, adaptive Gaussian point management, and a hybrid camera tracking algorithm to improve the accuracy and robustness of pose estimation. Extensive experiments demonstrate that DG-SLAM delivers state-of-the-art performance in camera pose estimation, map reconstruction, and novel-view synthesis in dynamic scenes, outperforming existing methods meanwhile preserving real-time rendering ability.

Main paper: 
 introduction: 
 Visual simultaneous localization and mapping (SLAM), the task of reconstructing a 3D map within an unknown environment while simultaneously estimating the camera pose, is recognized as a critical component to achieving autonomous navigation in novel 3D environments for mobile robots [1] . It has been widely used in various forms in fields such as robotics, autonomous driving, and augmented/virtual reality (AR/VR). However, the majority of previous research [2, 3, 4, 5, 6, 7, 8, 9, 10] typically relies on the assumption of static environments, limiting the practical applicability of this technology in daily life. Consequently, how to achieve accurate and robust pose estimation in dynamic scenes remains an urgent problem to be addressed for mobile robots.Recently, many researchers [6, 7, 8, 9, 10] have endeavored to replace the conventional explicit representations used in visual SLAM, such as Signed Distance Function (SDF), voxel grids [11] , meshes [12] , and surfel clouds [13] , with the neural radiance field (NeRF) [14] approach for reconstructing the neural implicit map. This novel map representation is more continuous, efficient, and able to be optimized with differentiable rendering, which has the potential to benefit applications like navigation and reconstruction. However, these methods exhibit two primary issues: the scene's bounds are required to be predefined to initialize the neural voxel grid; and the implicit representation proves challenging for information fusion and editing. To address these problems, recent works like GS-SLAM [15] , SplaTam [16] , and Gaussian splatting SLAM [17] leverage the 3D-GS [18] to explicit represent the scene's map. This explicit geometric representation is also smooth, continuous, and differentiable. Moreover, a substantial array of Gaussians can be rendered with high efficiency through splatting rasterization techniques, achieving up to 300 frames per second (FPS) at a resolution of 1080p. However, all these above-mentioned neural SLAM methods do not perform well in dynamic scenes. The robustness of these systems significantly decreases, even leading to tracking failures, when dynamic objects appear in the environment.To tackle these problems, we propose a novel 3D Gaussian-based visual SLAM that can reliably track camera motion in dynamic indoor environments. Due to the capability of 3D-GS to accomplish high-quality rendering in real-time, the SLAM system more readily converges to a global optimum during pose optimization, thereby achieving better and more stable pose optimization results. A cornerstone of our approach to achieving robust pose estimation lies in the innovative motion mask generation algorithm. This algorithm filters out sampled pixels situated within invalid zones, thereby refining the estimation process. In addition to the constraint of depth residual, we employ a spatiotemporal consistency strategy within an observation window to generate depth warp masks. By incrementally fusing the depth warp mask and semantic mask, the motion mask will become more precise to reflect the true motion state of objects. To improve the accuracy and stability of pose estimation, we leverage DROID-SLAM [19] odometry (DROID-VO) to provide an initial pose estimate and devise a coarse-to-fine optimization algorithm built upon the initially estimated camera pose. This aims to minimize the disparity between pose estimation and the reconstructed map, employing photorealistic alignment optimization through Gaussian Splatting. Moreover, this hybrid pose optimization approach effectively ensures the accuracy and quality of the generated depth warp mask, thereby facilitating better performance in the next camera tracking stage. To obtain high-quality rendering results, we propose a novel adaptive Gaussian point addition and pruning method to keep the geometry clean and enable accurate and robust camera tracking. Capitalizing on the factor graph structure inherent in DROID-SLAM, our system is capable of executing dense Bundle Adjustment (DBA) upon completion of tracking to eliminate accumulated errors.In summary, our contributions are summarized as follows: (i) To the best of our knowledge, this is the first robust dynamic Gaussian splatting SLAM with hybrid pose optimization, capable of achieving both real-time rendering and high-fidelity reconstruction performance. (ii) To mitigate the impact of dynamic objects during pose estimation, we propose an advanced motion mask generation strategy that integrates spatio-temporal consistent depth masks with semantic priors, thereby significantly enhancing the precision of motion object segmentation. (iii) We design a hybrid camera tracking strategy utilizing the coarse-to-fine pose optimization algorithm to improve the consistency and accuracy between the estimated pose and the reconstructed map. (iv) To better manage and expand the Gaussian map, we propose an adaptive Gaussian point addition and pruning strategy. It ensures geometric integrity and facilitates accurate camera tracking. (v) Extensive evaluations on two challenging dynamic datasets and one common static dataset demonstrate the state-of-the-art performance of our proposed SLAM system, particularly in real-world scenarios.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We have clearly stated the contribution and scope of the paper in the abstract.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Visual SLAM with dynamic objects filter. Dynamic object filtering is pivotal for reconstructing static scenes and bolstering the robustness of pose estimation. Existing approaches fall into two main categories: the first relies on re-sampling and residual optimization strategies to remove outliers, as seen in works such as ORB-SLAM2 [2] , ORB-SLAM3 [3] , and Refusion [5] . These methods, however, are generally limited to addressing small-scale motions and often falter in the face of extensive, continuous object movements. The second group employs the additional prior knowledge, for example, semantic segmentation or object detection prior [20, 21, 22, 23, 24, 25, 26] , to remove the dynamic objects. However, all these methods often exhibit a domain gap when applied in realworld environments, leading to the introduction of prediction errors. More recently, deep neural networks have been employed to build end-to-end visual odometry, which performs better in specific environments such as DROID-SLAM [19] , DytanVO [27] , and DeFlowSLAM [28] . However, these methods still require a substantial amount of training data and are unable to reconstruct the high-fidelity static map.RGB-D SLAM with neural implicit representation. Neural implicit scene representations, also known as neural fields [29] , have attracted considerable attention in the field of RGB-D SLAM for Keyframe selectionAdaptive point addition strategy$ %, ' = ) their impressive expressiveness and low memory footprint. Initial studies, including iMap [6] and DI-Fusion [30] , explored the utilization of a single MLP and a feature grid to encode scene geometries within a latent space. However, they both share a critical issue: the problem of network forgetting, which is catastrophic for long-term localization and mapping. In response to this limitation, NICE-SLAM [7] introduces a hierarchical feature grid approach to enhance scene representation fidelity and implements a localized feature updating strategy to address the issue of network forgetting. While these advancements contribute to improved accuracy, they necessitate greater memory consumption and may impact the system's ability to operate in real-time. More recently, existing methods like Vox-Fusion [10] , Co-SLAM [8] , and ESLAM [9] explore sparse encoding or tri-plane representation strategy to improve the quality of scene reconstruction and the system's execution efficiency. Point-SLAM [31] draws inspiration from the concept of Point-NeRF [32] , utilizing neural points to encode spatial geometry and color features. It employs an explicit method to represent spatial maps, effectively improving the accuracy of localization and mapping. All these methods have demonstrated impressive results based on the strong assumptions of static scene conditions. The robustness of these systems significantly decreases when dynamic objects appear in the environment. Recently, Rodyn-SLAM [33] proposed utilizing optical flow and semantic segmentation prior to filter out dynamic objects, and employing a neural rendering method as the frontend. However, this approach is computationally intensive and limits the maximum accuracy achievable in pose estimation.%,' ∈) * %' * -Π+(.%' ∘ Π + ,! (*%, '%)) -#$⋅ / 0$ " $# * = 234 min $! 8% 1 M ; #&% ' ( = > ?, $# ->) ⋅ / B# ⋅ C D# ( ( +8( 1 F) ; *∈," ( C G ?, $# -G) ⋅ / B# ⋅ C D# ( ( ℒ#$% = 1 ( ) &'( ) ( + , -,) ⋅ 0 (& * * ℒ)-./0 = 1 F) ; *∈," ( C G -G) ⋅ / B# ( ( ℒ1233!45 = #6ℒ758 + #9ℒ::!1+ #;ℒ<=3>?3D Gaussian splatting SLAM methods. Compared to the aforementioned NeRF-based SLAM methods, 3D Gaussian splatting (3D-GS) [18] has garnered widespread interest among researchers due to its advantages in high-fidelity and real-time rendering. Unlike previous implicit map representation methods, 3D-GS explicitly models scene maps by independent Gaussian spheres, naturally endowing it with geometric structure property. Some researchers [17, 15, 34, 16, 35] are exploring the replacement of implicit representations (NeRF) with 3D-GS in the mapping thread. However, these methods are currently constrained by the assumption of static environments, rendering them ineffective in dynamic scenes. It significantly restricts the practical application of Gaussian SLAM systems in real-life scenarios. Under the premise of ensuring high-fidelity reconstruction and realtime rendering, our system is designed to improve the accuracy and robustness of pose estimation under dynamic environments.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have discussed the limitations of the work in the conclusion part.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
approach: 
 Given a sequence of RGB-D frames {I i , D i } N i=1 , I t ∈ R 3 , D t ∈ R, our method (Fig. 1 ) aims to simultaneously recover camera poses {ξ i } N i=1 , ξ t ∈ SE(3) and reconstruct the static 3D scene map represented by 3D Gaussian sphere in dynamic environments. Similar to most modern SLAM systems [36, 37] , our system comprises two distinct processes: the tracking process as the frontend and the mapping process as the backend.To obtain real-time rendering and high-fidelity reconstruction mapping, we represent the scene as a set of 3d Gaussian ellipsoid G, which simultaneously possesses geometric and appearance properties.EQUATIONEach 3D Gaussian ellipsoid G i is composed of its center position µ i ∈ R 3 , covariance matrix Σ i ∈ R 3×3 , opacity α i ∈ R, and spherical harmonics coefficients h i ∈ R 16 .Color and depth splatting rendering. To obtain the rendering image of color and depth, we project the 3D Gaussian (µ w , Σ w ) in world coordinate to 2D Gaussian (µ I , Σ I ) on the image plane:EQUATION)where R is the viewing transformation and J is the Jacobian of the affine approximation of the projective transformation. Following the alpha blending method in 3DGS [18] , we accumulate the splatting Gaussian ellipsoid along the observation image pixel at the current estimation pose ξ i to render the color and depth as:EQUATIONwhere f i (•) is the Gaussian distribution function weighted by opacity α i . c i represents the color of the projected Gaussian point computed by learnable spherical harmonics coefficients h i . Similarly, d i denotes the depth of Gaussian point G i , obtained by projecting to z-axis in the camera coordinate.Accumulated opacity. We use accumulated opacity Ô to represent the rendering reliability for each pixel and judge whether the Gaussian map is well-optimized.EQUATIONFor each input keyframe, we select its associated keyframe set D within a sliding window. To reduce the computation and improve the accuracy of generating motion mask, we employ the depth warping operation solely on keyframes. To ensure the overlap between adjacent keyframes is not too small, we employ optical-flow distance to determine keyframe insertion. In regions with more intense motion, our goal is to insert as many keyframes as possible.For the pixel p in keyframe i, we reproject it onto keyframe j as follows:EQUATIONwhere K and T ji represent the intrinsic matrix and the transformation matrix between frame i and frame j, respectively. p homo i = (u, v, 1) is the homogeneous coordinate of p i .Given any associate keyframes D i , D j ∈ D, we utilize warp function f warp to compute the residual of reprojection depth value. By setting a suitable threshold e th , we derive the depth warp mask M wd j,i corresponding to dynamic objects as:EQUATIONwhere I m×n represents a matrix of the same size as the image, filled with ones. The ⊗ operation signifies that for each element in the matrix I m×n , we assess whether its warp depth meets a specified threshold and subsequently modify the corresponding value at that position. As illustrated in Fig. 1 , to derive a more precise warp mask, we consider the spatial coherence of object motion within a sliding window of length N and combine the multiple observation warp masks. Unlike ReFusion [5] ,our method can mitigate the potential impact of observation noise from a single warp mask. When object motion becomes significant, we only mask the foreground pixels where the depth residual is positive to avoid a large portion of pixels being masked as dynamic regions. Subsequently, we integrate the warp mask and semantic mask to derive the final motion mask M j as:EQUATIONThanks to our innovative approach to generating motion masks, the omission of dynamic objects by semantic priors can be effectively compensated. Furthermore, by leveraging a spatial consistency strategy, the inaccuracy of edge region recognition during depth warping can be significantly reduced.The constant speed motion model struggles to infer a reasonable initial optimized pose value in dynamic scenes. An inaccurate initial pose will lead to optimization getting trapped in local optima more easily and can affect the generation quality of the depth warp mask. To achieve more precise pose estimation in dynamic environments, we utilize the visual odometry (VO) component from DROID-SLAM [19] as the coarse pose estimation results in camera tracking. We also conduct a dense bundle adjustment in every interaction for a set of keyframes to optimize the corresponding pose G and depth d:EQUATIONwhere Σ ij = diag (w ij ), w ij represents the confidence weights. G ij denotes the relative transformation between the poses G i and G j . p i means a grid of pixel coordinates. Moreover, p * ij is corrected correspondence as predicted by updated optical flow estimation. To overcome the influence of dynamic objects on bundle adjustment, we introduce suppression through the motion mask M j applied to the weighted covariance matrix. Consequently, the weighted confidence associated with dynamic objects is reduced to zero.To further improve the accuracy of pose estimation and minimize inconsistencies between the estimated pose and the reconstructed map, we implement fine camera tracking by leveraging Gaussian splatting, based on the initial obtained pose. Due to obstructions by dynamic objects and the inadequate optimization of Gaussian points in the map, the rendered image may exhibit blurring or black floaters. Therefore, we employ the accumulated opacity, as outlined in 4, to denote the pixel rendering reliability. The reliable mask O i for camera tracking is generated as follows:EQUATIONwhere µ j , v j represents the pixel location. When the accumulated opacity at the pixel exceeds a given threshold τ track , we consider the Gaussian point associated with that pixel to be well-optimized. Consequently, using the rendered image at these pixels for pose estimation is deemed reliable. The overall loss function is finally formulated as the following minimization:EQUATION)Where ξ i denotes the camera pose requiring optimization. C and D denote the ground truth color and depth map, respectively. M represents the number of sampled pixels in the current image. Note that only pixels with valid depth value N d are considered in optimization.This hybrid pose optimization approach effectively ensures the accuracy and quality of the warp mask, thereby facilitating better performance in the next camera tracking stage. In short, this hybrid pose optimization strategy enables us to achieve more precise and robust pose estimation whether in dynamic or static environments.Map initialization. For the first frame, we do not conduct the tracking step and the camera pose is set to the identity matrix. To better initialization, we reduce the gradient-based dynamic radius to half so that can add more Gaussian points. For pixels located outside the motion mask, we sample and reproject them to the world coordinates. We initialize the Gaussian point color and center position µ i with the RGB value and reprojection coordinate of the pixel, respectively. The opacity α i is set as 0.1 and the scale vector S i is initialized based on the mean distance of the three nearest neighbor points. The rotation matrix R i is initialized as the identity matrix.Map optimization. To optimize the scene Gaussian representation, we render depth and color in independent view as seen Eq. 3, comparing with the ground truth map:EQUATIONIn contrast to existing methods, we introduce the motion mask M i to remove sampled pixels within the dynamic region effectively. The final Gaussian map optimization is performed by minimizing the geometric depth loss and photometric color loss :EQUATIONwhere L ssim denotes the structural similarity loss between two images. Moreover, λ 1 , λ 2 , λ 3 are weight factors for balance in the optimization process.Adapative Gaussian point adding strategy. To guarantee the superior scene representation capability of 3D Gaussian, we also employ dynamic Gaussian point density, which is inspired by Point-SLAM [31] . The insertion radius is determined based on color gradient, allowing for a denser allocation of points in areas with high texture while minimizing point addition in regions of low texture. This method efficiently manages memory consumption by adapting the point density according to the textural complexity of the scene.To ensure the points we add are both necessary and effective, we adopt a two-stage adaptive point-add strategy. For new viewpoints without previous observation, we initially perform uniform sampling across the entire image to ensure that new observed areas can be covered by Gaussian points. Moreover, if the accumulated opacity Ô, calculated by Eq. 4, falls below the threshold o th , or the depth residual between the rendered pixels and the ground truth depth is excessively large, we then add 3D Gaussian points based on these under-fitting pixels. At last, these new Gaussian points will be initialized based on the point density.Map point deleting strategy. Given that the added 3D Gaussian points have not been subjected to geometric consistency verification and may exhibit unreasonable representation values during optimization, this could lead to the generation of a low-quality dense map or the introduction of numerous artifacts in the rendering image. we implement the pruning operation as part of the Gaussian map optimization. To ensure the points we delete are both reasonable and accurate, we also adopt a two-stage point delete strategy. For all Gaussian points observed from the current viewpoint, we delete points based on three criteria: the opacity value, the maximum scale, and the ratio between the ellipsoid's major and minor axes, described as follow:EQUATIONMoreover, due to potential inaccuracies along the edges of the current motion mask, adding these points could result in artifacts within the scene map. Thus, we project these points on keyframes in a small sliding window to recheck whether they can be observed by these keyframes. If the number of observations of a point is too low, we consider its addition to be insufficiently robust, and therefore, it will be removed from the current Gaussian map.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 Datasets. Our methodology is evaluated using three publicly available challenging datasets: TUM RGB-D dataset [38] , BONN RGB-D Dynamic dataset [5] and ScanNet [39] . These three datasets contain both challenging dynamic scenes and real static scenes. This selection facilitates a comprehensive assessment of our approach under varied conditions, demonstrating its applicability and robustness in real-world indoor environments. Acc. [ Instances of tracking failures are denoted by "X". The most superior outcomes within the domain of RGB-D SLAMs are highlighted in bold for emphasis.Metrics. For the evaluation of pose estimation, we utilize the Root Mean Square Error (RMSE) and Standard Deviation (STD) of Absolute Trajectory Error (ATE) [38] . Prior to assessment, the estimated trajectory is aligned with the ground truth trajectory via Horn's Procrustes method. [40] , ensuring a coherent basis for evaluation. To evaluate the reconstruction quality of static maps in dynamic scenes, we employ three metrics(i) Accuracy (cm), (ii) Completion (cm), and (iii) Completion Ratio (percentage of points within a 5cm threshold), following NICE-SLAM [7] . Since the BONN dataset provides only the ground truth point cloud, we randomly sampled 200,000 points from the GT point cloud and the reconstructed mesh surface to calculate these metrics.Implementation details. We run our DG-SLAM on an RTX 3090 Ti GPU at 2 FPS on BONN datasets, which takes roughly 9GB of memory. We set the loss weight λ 1 = 0.9 , λ 2 = 0.2 and λ 3 = 0.1 to train our model. The number of iterations for the tracking and mapping processes has been set to 20 and 40, respectively. For the Gaussian points deleting, we set τ α = 0.005, τ S1 = 0.4 and τ S2 = 36 to avoid the generation of abnormal Gaussian points. What's more, we utilize Oneformer [41] to generate prior semantic segmentation. For the depth wrap mask, we set the window size to 4 and the depth threshold to 0.6. We also adopt the keyframe selection strategy from DROID-VO [19] based on optical flow. We evaluated our method on the balloon and move_no_box2 sequences of the BONN dataset to show the qualitative results of the generated motion mask. In these two sequences, in addition to the typical motion of pedestrians, there are movements of atypical objects accompanying the human, such as balloons and boxes. It might be missed if we rely solely on the semantic prior. As shown in Fig. 2 , our generated methods notably improve the precision of motion mask segmentation, effectively reducing the inconsistencies along edge regions and detecting the true dynamic objects.To more effectively showcase the performance of our system in dynamic environments, we evaluate the reconstruction results from both qualitative and quantitative perspectives. Given dynamic scene datasets seldom offer static GT mesh or point cloud, we utilize the BONN dataset for our quantitative analytical experiments. We compare our DG-SLAM method against current state-of-the-art neuralbased SLAM methods, all of which are available as open-source projects.As shown in Tab. 1, our method significantly surpasses contemporary approaches in terms of accuracy, completion, and completion ratio metrics, achieving state-of-the-art performance. Meanwhile, the reconstructed static map can be rendered with high fidelity, as shown in Fig. 3 . This also indirectly demonstrates that our methods can generate a more accurate static map compared with other mainstream SLAM systems.To comprehensively assess the tracking performance of our DG-SLAM approach, we conduct comparative analyses within highly dynamic, slightly dynamic, and static environments. The comparison methods encompass classical SLAM methods such as ORB-SLAM3 [3] , ReFusion [5] ,MIDfusion [43] , and EM-fusion [44] , and widely recognized NeRF-based SLAM systems such as NICE-SLAM [7] , iMap [6] , Vox-Fusion [10] , ESLAM [9] , Co-SLAM [8] . We also incorporate a comparison with the newly proposed dynamic neural RGB-D SLAM system, Rodyn-SLAM [33] . Furthermore, our evaluation extends to the latest advancements in 3D Gaussian-based SLAM, including SplaTAM [16] and GS-SLAM [17] .Dynamic scenes. As shown in Tab. 2, we report the results on three highly dynamic sequences, two slightly dynamic sequences, and one static sequence from the TUM RGB-D dataset. Our system exhibits exceptional tracking performance, attributed to the implementation of the map point deleting strategy and the powerful coarse-to-fine camera tracking algorithm. Furthermore, the tracking capabilities of our system have also been rigorously evaluated on the intricate and demanding BONN RGB-D dataset, with outcomes presented in Tab. 3. In dynamic scenarios characterized by heightened complexity and challenge, our method has consistently demonstrated superior performance, underscoring its effectiveness and reliability in real-world navigation applications. Meanwhile, we also showcased the number of iterations of the tracking and mapping process, taking TUM as an example, as shown in Tab. 5. Compared to other methods, our method achieved superior results while maintaining competitive iterations and efficiency.Static scenes. To better illustrate the robustness of our system, we also evaluate our methods with existing SLAM on common real-world static sequences from ScanNet [39] . As shown in Tab. 4, our DG-SLAM still achieves competitive performance within static scenes with fewer tracking and mapping iterations, despite our method being designed for dynamic scenes. Notably, the motion mask will become irrelevant in static scenes. Thus, it can sufficiently demonstrate the effectiveness of our proposed hybrid camera tracking strategy and adaptive Gaussian point management strategy. [16] 200 30 GS-SLAM [17] 100 81 DG-SLAM(Ours) 20 40 Table 5 : Running iterations on TUM.To evaluate the efficacy of the proposed algorithm in our system, we conducted ablation studies across seven representative sequences from the BONN dataset. Given that the semantic priors within the TUM dataset have covered major motion categories (humans) while motion categories in BONN contained undefined dynamic objects such as balloons and boxes, thus we opted to perform ablation studies on the BONN dataset. We calculated the average ATE and STD metrics to illustrate the impact of various components on the overall system performance. As the results presented in Tab. 6, the findings affirm the effectiveness of all proposed methods in improving camera tracking capabilities. Specifically, the strategies for adding and pruning points are equally crucial, significantly impacting the quality of the Gaussian map reconstruction and, in turn, affecting the tracking performance. The depth warp operation effectively removes floaters from multiple viewpoints, thereby noticeably enhancing the quality of the rendered images. One of the main contributions comes from the hybrid camera tracking strategy, which indirectly underscores the importance of eliminating inconsistencies between pose estimation and map reconstruction.As shown in Tab. 7, we report time consumption (per frame) of the tracking and mapping without computing semantic segmentation. These results were achieved through an identical experimental setup that involved conducting 20 iterations for tracking and 40 iterations for mapping, all processed on an RTX 3090Ti GPU. Benefiting from the rapid execution speed of Droid-VO and fast rendering of 3D Gaussian Splatting, our method exhibits a leading edge in terms of time consumption during the tracking process. Although our approach does not match the mapping speed of Co-SLAM, it achieves high-quality mapping and high-fidelity rendering with a competitive mapping running time. We also evaluate the inference time of our used semantic segmentation network, which required 163ms for every frame. It should be noted that our approach does not focus on the specific semantic segmentation network used, but rather on the fusion method itself.  
conclusion: 
 In this paper, we have presented DG-SLAM, a robust dynamic Gaussian splatting SLAM with hybrid pose optimization under dynamic environments. Via motion mask filter strategy and coarse-to-fine camera tracking algorithm, our system significantly advances the accuracy and robustness of pose estimation within dynamic scenes. The proposed adaptive 3D Gaussians adding and pruning strategy effectively improves the quality of reconstructed maps and rendering images. We demonstrate its effectiveness in achieving state-of-the-art results in camera pose estimation, scene reconstruction, and novel-view synthesis in dynamic environments. While the tracking and reconstruction of largescale scenes is currently the biggest limitation of our system, we believe it will be addressed by a more flexible loop-closure optimization algorithm in future work. Moreover, the accuracy of pose estimation of our system is still influenced by the segmentation precision of the semantic prior. Therefore, efficiently perceiving moving objects within the dynamic scenes remains an unresolved issue that warrants further exploration.NeurIPS Paper Checklist 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed the societal impacts of this paper in the introduction and method parts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper has not released data or models that have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the assets used in the paper are properly licensed. Our paper cites the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets, including data and models. We also did not publish or submit the code.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The research topic of this paper is robotic SLAM, not involving crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The research topic of this paper is robotic SLAM, not involving crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This paper proposes a DG-SLAM system in dynamic environments grounded in 3D Gaussians with fidelity reconstructions. The key parts include motion mask generation, adaptive Gaussian point management, and a hybrid camera tracking algorithm to improve the accuracy and robustness of pose estimation. Experimental results show the effectiveness of the method. 

The contribution of the paper is to tackle dynamic objects when appearing in the environment. The existing neural SLAM methods do not perform well in dynamic scenes. The robustness of these systems significantly decreases, even leading to tracking failures. This paper proposes a 3D Gaussian-based visual SLAM that can reliably track camera motion in dynamic indoor environments.

The paper received one “Reject” rating, one “Weak Accept” rating, and two “Borderline Accept” ratings. 

Reviewer UDnM giving “Reject” proposed a series of questions. The questions were addressed or discussed by the authors. In particular, the experiments were performed with classical methods Co-Fusion, MID-Fusion, EM-Fusion on accuracy in Tables 2 and 3, and performed with Co-Fusion, MID-Fusion on runtime. The authors discussed both NeRF and GS-based SLAM methods exhibit a certain gap in runtime when compared to traditional approaches. Addressing this discrepancy is a challenge that the research community continues to actively pursue.

Reviewer DPxX thinks the authors have processed all his/her concerns. Reviewer DPxX keeps the “Weak Accept”.

Reviewer MuLC thinks his/her concerns are either reasonably addressed or considered as future works. Reviewer MuLC raised the score to Borderline Accept.

Reviewer fGfv didn’t give replies to the authors’ rebuttals. Reviewer fGfv’s score is “Borderline Accept”.

In summary, the paper has new contributions and the questions of reviewers are addressed in the authors’ replies. 

An expected revision is to see the experiments compared with ORBSLAM3 on accuracy. Comparisons on runtime are not required. ORBSLAM3 is also a dense RGBD SLAM and can deal with dynamic objects. Generally, many SLAM systems can deal with dynamic objects by removing the matchings on the dynamic objects. When the occupancies of the dynamic objects on images are larger, the camera pose tracking will fail. How about the proposed method at the time? 

Another expected revision is to see the discussions on why the method can’t be used in outdoor environments. Although RGBD cameras fail to capture depths, Lidar-cameras still can be used.

Reviews: review: paper_summary: paper_summary: The paper combines existing methods for deep SLAM (DROID-SLAM) and 3D Gaussian splatting for 3D mapping into a system and adds an approach for dynamics filtering by depth warping to make tracking more robust in dynamic scenes. The approach is evaluated on several dynamic SLAM benchmarks and compared with recent deep learning based RGB-D SLAM approaches.
strength_weakness: strength_weakness: - Unfortunately, this paper has significant shortcomings in writing quality and clarity. Many variables and concepts are not defined properly (for instance, J, R in eq 2., \otimes operator, I_{m\times n}, \Pi in eq 8, S_i in l. 192, the symbols in eq. 13). 
- Some phrases in the introduction are not accurate or proper English. What does "finishing pose estimation robustness" mean? What are invalid zones, what does furnishing an initial pose estimate mean?
- Sec. 2, the first paragraph starts with the heading "traditional visual SLAM with dynamic objects filter", but contains recent deep SLAM approaches for static scenes. Please restructure.
- The raycasted color and depth in eq. 3 requires a specific ordering of the gaussian splats, as indicated in the products ranging from j=1 to i-1. Please explain how it is established.
- What does the intersection operation over pixels achieve in eq 6 and 9? What does it mean to apply an indicator function and take a \otimes product with the I_mxn ? What is I_mxn? 
- The variable \mu is used for coordinates and means of 3D Gaussians ambiguously, please change notations.
- l. 179, how is the camera pose parametrized and optimized ?
- l. 188, what is the dynamic radius?
- l. 192, what is the scale vector S_i and why should it be computed from the mean distance of three nearest neighbor points? Where do these points come from?
- eq 12, how are the lambda_1-3 chosen?
- l. 235, "unit quaternion algorithm" is not the right name for the method but rather Horn's procrustes method.
- l. 257, what is an "analytical experiment" ?
- Table 2, 3, what are the metrics for the numbers ? ATE, RPE ? how is it computed?
- Tables 2, and 3 should also compare with classical dynamic SLAM methods like Co-Fusion [*1], MID-Fusion [*2], EM-Fusion [*3] to name a few. Also these methods are missing from the discussion of related work.
- what is an "invalid Gaussian point filter" in l. 272 ?
- Table 5 seems to indicate that different settings of the iteration count hyperparameter are needed for the various datasets. This indicates an issue with generalization for the method. Are other hyperparameters chosen consistently across datasets?
- l. 290 what does to "encompass major motion categories" mean ?
- l. 291, what is the definition of the STD metric ?
- Table 7 / l. 302, please include run-time for semantic segmentation as it seems essential according to Table 6.
- How is the semantic segmentation mask obtained ? What is the runtime of this processing step?
- The paper does not discuss limitations and assumptions of the method beyond not performing loop closing.

[*1] Martin Ruenz, Lourdes Agapito. Co-fusion: Real-time segmentation, tracking and fusion of multiple objects. ICRA 2017

[*2] Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew J. Davison, Stefan Leutenegger. MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM. ICRA 2019

[*3] Michael Strecke, Joerg Stueckler. EM-Fusion: Dynamic Object-Level SLAM With Probabilistic Data Association. ICCV 2019
questions: questions: See questions in paper weaknesses.
limitations: limitations: The conclusion provides an obvious limitation of the approach due to not considering loop closing. Further limitations and assumptions are not discussed.
paper_summary: paper_summary: The paper present DG_SLAM to address the inconsistent observation of geometry and photometry in dynamic environments among the current slam-related field.  Specifically, the authors develop several technologies such as the motion mask generation, adaptive gaussian point management, and a hybrid camera tracking algorithm to improve the performance of slam system. And the experiments on TUM RGB_D and BONN RGB_D dataset showcases the state-of-the-art performance both on static and dynamic environments.
strength_weakness: strength_weakness: 1.	A motion mask generation strategy is used to mitigate the potential impact of observation noise from a single warp mask, can the authors give more details about this part? Are there any theoretically analysis for that?

2.	A hand-crafted rule is adapted in the map point deleting stage. For example, the authors delete points based on three baselines: the opacity value, the maximum scale, the ration between ellipsoid’s major and minor axes, can authors give more explanation about these criteria? Moreover, the authors showcase the removement of number of observations of the point that are too low, but fail to show its quantitative criteria.
3.	the running time on BONN dataset during the mapping phase seems a bit large in table 7, can authors give several details about this result? Are there any connections with the proposed map point deleting strategy?
questions: questions: see weakness part
limitations: limitations: The method fails to be conducted on several large-scale real scenes.
paper_summary: paper_summary: The paper introduces the first robust dynamic visual SLAM system based on 3D Gaussian Splatting. It provides precise camera pose estimation and high-fidelity reconstructions. Strategies such as motion mask generation, adaptive Gaussian point management, and hybrid camera tracking are proposed to improve the accuracy and robustness of pose estimation.
strength_weakness: strength_weakness: 1. The evaluation conducted in the paper primarily compares the proposed method against NeRF or Gaussian-splatting-based SLAM systems, which is not sufficient. It would be beneficial to include traditional SLAM systems as baselines, which present a more robust tracking capability, such as ReFusion [1] and ORB-SLAM3 [2].

2. Lacks the comparison with MonoGS [3] as the baseline of GS-based SLAM. MonoGS demonstrates more advanced map reconstruction ability compared with other GS-based alternatives, and presents robust camera tracking utilizing Jacobian optimization (e.g. compared with trivial pose optimization via photometric loss implemented in SplaTAM). It should be included as a baseline method for more exhausted comparisons.

3. Lacks survey on the literature of related work, such as Crowd-SLAM [4] and DDN-SLAM [5], which were also proposed for the same motivation of reconstruction on the dynamic environment.


4. The experiments on two datasets have shown the effectiveness (see point 5) of the proposed method in dynamic indoor settings. However, for a SLAM system, it would be preferable to conduct experiments on real-world scenes, especially in outdoor environments.

5. The proposed system largely relies on dynamic object removal in the tracking system; however, it lacks certain optimization for the mapping system. This results in the reconstructed scenes still containing artifacts.


[1] Palazzolo, J. Behley, P. Lottes, P. Giguere, and C. Stachniss, “Refusion: 3d reconstruction
in dynamic environments for rgb-d cameras exploiting residuals,” in IROS, 2019.

[2] C. Campos, R. Elvira, J. J. G. Rodr ́ıguez, J. M. Montiel, and J. D. Tard ́os, “Orb-slam3: An
accurate open-source library for visual, visual–inertial, and multimap slam,” IEEE TRO, 2021.

[3] Matsuki, Hidenobu, et al. "Gaussian splatting slam." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Soares, João Carlos Virgolino, Marcelo Gattass, and Marco Antonio Meggiolaro. "Crowd-SLAM: visual SLAM towards crowded environments using object detection." Journal of Intelligent & Robotic Systems 102.2 (2021): 50.

[5] Li, Mingrui, et al. "Ddn-slam: Real-time dense dynamic neural implicit slam with joint semantic encoding." arXiv preprint arXiv:2401.01545 (2024).
questions: questions: 1. According to Table 6, the segmentation mask seems to play a much more significant role than other components, such as the depth warp mask. Please discuss the utility of the other components in detail. A concern here is that a more accurate segmentation mask (e.g. produced by SAM) might facilitate even better dynamic removal without the need for other components.

2. In the implementation detail, the author should elaborate more on how to perform semantic segmentation, the parameter of depth warping, and the keyframe selection strategy they have integrated in the proposed system.

3. The dynamic features are removed in the tracking system; however, a large portion of the removed features might reduce the tracking accuracy. In other words, to what extent can the method proposed in the paper tolerate motion? Is there any failure case/example of tracking, e.g. when dynamic features dominate the current view?

4. The semantic mask is retrieved by utilizing a pretrained segmentation model. There is a concern regarding the generalizability of this semantic mask. How is the dynamic object defined with respect to the semantic mask? For instance, the human might be considered a dynamic object in the BONN dataset, but might be stable in other scenes (e.g. wax figure). In addition, how does this integration of the semantic model affect the real-time processing capability of the SLAM system?
limitations: limitations: There is no societal impact of this work, and limitations are discussed in the paper.
paper_summary: paper_summary: This paper proposed a dynamic RGB-D SLAM system based on 3D Gaussian Splatting (3D-GS) and DROID-SLAM. Semantic segmentation masks and depth warping residuals are used to generate motion masks to remove the dynamic part of the scene. Experiments on three real-world datasets show the proposed approach achieves superior results on both dynamic and static scenes.
strength_weakness: strength_weakness: 1. The technical novelty is a bit limited. The paper focuses on adding new components to previous SLAM methods in order to achieve dynamic SLAM.
2. The hybrid coarse-to-fine camera tracking seems like a trick: The coarse stage directly comes from DROID-SLAM while the fine stage simply adds masks to 3DGS-based tracking.
questions: questions: Please see Weakness.
limitations: limitations: Yes.

Comments: title: Response to Reviewer UDnM [2/2]
Response to Reviewer UDnM [1/2]
comment: Dear Reviewer UDnM,

We appreciate your time for reviewing, and thanks again for the valuable comments and suggestions. As the discussion phase is nearing its end, we wondered if the reviewer might still have any concerns that we could address. We believe our responses on consistent comparison, rotation representation, running time, and limitations addressed all the questions/concerns, and we hope that our work’s impact and results are better highlighted with our responses. 

It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you！

Best wishes,

Authors
### Q4: Running time issue.
A4: It should be noted that our approach does not focus on the specific semantic segmentation network used, but rather on the fusion method itself. All the baseline methods in Table 7 in our main paper do not include the time for semantic segmentation. Semantic segmentation is typically employed as a preprocessing operation and serves as an input to the system. Most of the dynamic SLAM methods also did not include the time for semantic segmentation instead including it as part of the preprocessing step. Consequently, Table 7 captures the total runtime of our SLAM system, and the time taken for semantic segmentation is not included. Considering the reviewer's concerns about the runtime of the semantic segmentation network we utilized, we have tested the inference time of the Oneformer [7] method, which is 163ms for every frame. Correspondingly, the running time including semantic segmentation is 245.57ms for tracking.

To avoid potential ambiguity, we will provide a detailed description of the running time per frame for the semantic segmentation method employed in the revised paper.

Note that DG-SLAM is not optimized for real-time operation. As the first system to propose ***dynamic*** Gaussian splatting for SLAM, this paper primarily focuses on designing an effective approach to achieve robust tracking and mapping. With ongoing advancements in these research fields and improvements in computing power, the processing speeds for semantic segmentation are expected to increase, ensuring they do not become bottlenecks for our method. 

### Q5: Discuss more limitations and assumptions.
A5: Our approach has a certain tolerance for the accuracy of semantic segmentation. When the semantic segmentation model generates incomplete or incorrect segmentation masks for fewer frames, our system can still perform accurate and robust camera tracking. This is attributed to the coarse-to-fine pose optimization strategy where the motion estimation from the dense optical flow can tolerate some segmentation failures. Thus the coarse stage can provide a robust initial pose for the fine stage.

Even when addressing complex pose estimation issues in dynamic scenes, the runtime of our mapping process is approximately the same as that of other GS-based SLAM methods, as shown in Table 2 (one-page global PDF). The system runtime reported in the original paper was evaluated on a 3090ti GPU. With ongoing advancements in GPU computational capabilities, we believe our method is capable of real-time computation, enabling the live processing of sensory data streams.

Hope our response helps the reviewer's final recommendation. Thank you!
### Q1: Notation issue.
A1: We are grateful for the reviewer's valuable comments. We will revise and refine the notations in the revision to eliminate any potential ambiguities.

### Q2: Rotation representation and constraints within the SO(3) group.
A2: We utilize the quaternion as the rotation representation in the optimizer to facilitate pose optimization. Throughout the optimization process for the rotational variables, we compute the derivative of the final loss directly with respect to the current quaternion representation. This chain rule differentiation process is structured into two stages. In the first stage, the derivative from the quaternion to the rotation matrix is calculated using automatic differentiation within Pytorch, which allows for the automatic capture of optimization variables. In the second stage, the derivative of the final loss with respect to the rotation matrix is manually computed using the Jacobian matrix. This calculation is implemented using CUDA code during the Gaussian rendering process. Upon obtaining the optimized quaternion, we normalize it to achieve a unit quaternion. This entire optimization process ensures compliance with the constraints on the SO(3) group.

### Q3: More experiments and consistent comparison.
A3: Thank you for your suggestions. To address the reviewer's concern, we provide the running time of tracking and mapping on the TUM dataset, as shown in Tab. 1. All the experiments were conducted with 20 iterations for tracking and 40 iterations for mapping. We have also added a comparison with the classical methods: Co-Fusion and MID-Fusion, where the running time is cited from the original papers.

**Table1: Run-time comparison on TUM f3_walk_static. * denotes including semantic segmentation time.**
||Tracking[ms]| Mapping[ms]|Avg[ms]|
|:----:|:----:|:----:|:----:|
|Co-Fusion|-|-|**83.3**|
|MID-Fusion *|-|-|400.0|
|NICE-SLAM|3186.2|1705.1|4892.3|
|E-SLAM|2045.9|1641.4|3688.5|
|Point-SLAM|2279.5|1544.4|3823.9|
|Co-SLAM|101.4|**140.1**|241.6|
|Ours|**89.2**|549.3|645.9|

Classical methods utilize an efficient variant of the Iterative Closest Points (ICP) algorithm for camera tracking and avoid a training process for mapping. Moreover, the implementation of classical methods has been optimized through the use of a C++ framework, resulting in faster runtime compared to neural SLAM methods. Currently, both NeRF and GS-based SLAM methods exhibit a certain gap in runtime when compared to traditional approaches. Addressing this discrepancy is a challenge that the research community continues to actively pursue.
Dear Reviewer DPxX

We appreciate the reviewer's time for reviewing and thanks again for the valuable comments.

Best wishes

Authors
Dear Reviewer MuLC

We appreciate the reviewer's time for reviewing and thanks again for the valuable comments.

Best wishes

Authors
Thanks for the detailed response to my comments.

Some further comments/questions:
* Even if the methods have been introduced in previous papers, used notation needs to be defined properly and in a self-contained way in a scientific paper to avoid any ambiguities. The paper should be revised accordingly.
* A7, which rotation representation is used for optimization and how are the constraints on SO(3) handled?
* A16, Table 7 results are from a different dataset than Table 5. Run-time results on the TUM datasets would be needed to relate the iterations with run-time. Please also include comparison with the classical methods [4-6]
* A19, the semantic segmentation is essential for the system and therefore the runtime cannot be omitted. Please include the timing in the run-time evaluation. Do the baseline methods in Table 7 also contain semantic segmentation? Is it counted into the run-time?
* A21, please also discuss limitations and assumption wrt. the design choices of the proposed method. For instance, how does the approach depend on the accuracy of semantic segmentation? Discuss if live processing of sensory streams is possible with this approach.
Dear Reviewer  fGfv, 

Many thanks for your support. Could you please read this rebuttal and the below? Then give your responses? And any discussions are welcome. 

AC
I appreciate the response from the authors, and my concerns are either reasonably addressed or considered as future works. Hence I raise my score to Borderline Accept.
Thanks, the authors have processed all my concerns. I keep my rate unchanged.