GS$^{3}$LAM: Gaussian Semantic Splatting SLAM

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 10:33:21

Abstract: Recently, the multi-modal fusion of RGB, depth, and semantics has shown great potential in the domain of dense Simultaneous Localization and Mapping (SLAM), as konwn as dense semantic SLAM. Yet a prerequisite for generating consistent and continuous semantic maps is the availability of dense, efficient, and scalable scene representations. To date, existing semantic SLAM systems based on explicit scene representations (points/meshes/surfels) are limited by their resolutions and inabilities to predict unknown areas, thus failing to generate dense maps. Contrarily, a few implicit scene representations (Neural Radiance Fields) to deal with these problems rely on time-consuming ray tracing-based volume rendering technique, which cannot meet the real-time rendering requirements of SLAM. Fortunately, the Gaussian Splatting scene representation has recently emerged, which inherits the efficiency and scalability of point/surfel representations while smoothly represents geometric structures in a continuous manner, showing promise in addressing the aforementioned challenges. To this end, we propose $\textbf{GS$^3$LAM}$, a $\textbf{G}$aussian $\textbf{S}$emantic $\textbf{S}$platting $\textbf{SLAM}$ framework, which takes multimodal data as input and can render consistent, continuous dense semantic maps in real-time. To fuse multimodal data, GS$^3$LAM models the scene as a Semantic Gaussian Field (SG-Field), and jointly optimizes camera poses and the field by establishing error constraints between observed and predicted data. Furthermore, a Depth-adaptive Scale Regularization (DSR) scheme is proposed to tackle the problem of misalignment between scale-invariant Gaussians and geometric surfaces within the SG-Field. To mitigate the forgetting phenomenon, we propose an effective Random Sampling-based Keyframe Mapping (RSKM) strategy, which exhibits notable superiority over  local covisibility optimization strategies commonly utilized in 3DGS-based SLAM systems. Extensive experiments conducted on the benchmark datasets reveal that compared with state-of-the-art competitors, GS$^3$LAM demonstrates increased tracking robustness, superior real-time rendering quality, and enhanced semantic reconstruction precision. To make the results reproducible, the source code will be publicly released.

Main paper: 
 introduction: 
 By integrating semantic understanding into map, semantic Simultaneous Localization and Mapping (SLAM) achieves simultaneous estimation of camera poses while constructing maps that maintain consistency across geometry, appearance, and semantics. In comparison to conventional SLAM techniques, it excels in the identification, classification, and correlation of entities within scenes. Nowdays, semantic SLAM systems have been applied in various domains, such as robotics [1, 13] and autonomous driving [1, 10, 20] .To date, existing semantic SLAM systems based on explicit scene representations often resort to points/surfels [15, 23, 26, 28] , grids [16] , or voxels [8, 11, 17] to construct maps. Although these representations offer advantages in geometry, storage, computational efficiency, and scalability, they face challenges in predicting unknown regions and are constrained by limited resolutions, thus being unable to generate dense semantic maps. Contrarily, recent emerging neural rendering techniques based on implicit scene representations, such as Neural Radiance Fields (NeRF) [14] , have shown potentials to deal with these challenges. NeRF portrays scenes as continuous implicit volume functions, enabling realistic novel view synthesis with minimal storage requirements. Based on it, several studies [3, 32] incorporate additional MLP channels to encode and decode semantic labels, while jointly optimizing camera poses and semantic scenes. However, due to the computationally expensive ray tracing-based volume rendering technique of NeRF, these methods fail to meet the real-time demands of SLAM.Fortunately, we observe the emergence of 3D Gaussian Splatting (3DGS) [7] , which demonstrates exceptional capabilities in dense 3D reconstruction. This method represents the scene as dense Gaussian clouds and achieves efficient rendering through tile-based rasterization. We show that 3DGS has great potential in addressing the aforementioned challenges. As a semantic SLAM scene representation, it inherits the efficiency, locality, and modifiability of point/surfel representations while smoothly and differentiably representing the geometric structure in a continuous manner, enabling the reconstruction of rich and complex details in dense maps. To further improve the capabilities of semantic SLAM in tracking, rendering, and semantic reconstruction, it is a natural idea to extend 3DGS as a semantic scene representation, but surprisingly such a simple idea has seldom been explored in existing literature. In this work, based on the above-mentioned findings, we propose a dense semantic SLAM framework, GS 3 LAM (Gaussian Semantic Splatting SLAM), to fully leverage the advantages of 3DGS.However, the effective embedding and real-time optimization of high-dimensional semantic categories pose profound challenges for GS 3 LAM. To deal with these issues, GS 3 LAM models the scene as a Semantic Gaussian Field (SG-Field), wherein semantic categories are represented as low-dimensional implicit features. By means of a simple decoder, GS 3 LAM efficiently transforms these features into semantic categories, facilitating the conversion between 3D implicit features and 2D semantic labels.Furthermore, within the SG-Field, irregular Gaussian scales hinder the accurate representation of geometric surfaces, making it unacceptable for pixel-level semantic reconstruction. To address this issue, we propose a Depth-adaptive Scale Regularization (DSR) strategy. This strategy constrains scales within a depth-dependent range, indirectly aligning Gaussians with geometric surfaces, thus can effectively reduce blurring on object surfaces and enhance both tracking robustness and semantic reconstruction accuracy. Finally, to address the forgetting phenomenon in GS 3 LAM, we propose a Random Sampling-based Keyframe Mapping (RSKM) strategy, which proves to be more effective than the Local Covisibility Keyframe Mapping (LCKM) strategy commonly adopted in 3DGS-based SLAM systems. Our observation suggests that the latter method introduces a considerable bias during the optimization of the Gaussian field, thereby leading to poor global map consistency. In particular, as depicted in Fig. 2 (a), frames with dense co-observations (dense camera trajectories) and increased optimization iterations (large point radii) exhibit lower PSNR values (darker color), suggesting challenges in achieving convergence of the Gaussian field under the LCKM strategy. Conversely, as shown in Fig. 2 (b), our proposed RSKM strategy not only enhances the rendering quality of the global map (higher mean PSNR, 𝜇 𝑃𝑆𝑁 𝑅 ) but also ensures high consistency among all perspectives (smaller PSNR variance, 𝜎 𝑃𝑆𝑁 𝑅 ), effectively reducing the optimization bias.Our contributions are summarized as follows:(1) As depicted in Fig. 1 , GS 3 LAM is a Gaussian Splatting Semantic SLAM framework, which models the scene as a Semantic Gaussian Field (SG-Field) to efficiently facilitate the conversion between 3D semantic features and 2D labels. By the joint optimization of camera poses and field for appearance, geometry, and semantics, it achieves robust tracking, real-time high-quality rendering, and precise semantic reconstruction. (2) A Depth-adaptive Scale Regularization (DSR) scheme is proposed to reduce the blurring of geometric surfaces induced by irregular Gaussian scales within the SG-Field. By constraining Gaussian scales within a reasonable range determined by depth, it alleviates the ambiguity of geometric surfaces, thereby enhancing accuracy in semantic reconstruction. (3) To address the forgetting phenomenon in GS 3 LAM, we propose an efficacious Random Sampling-based Keyframe Mapping (RSKM) strategy, which exhibits notable superiority over prevalent local covisibility optimization strategies commonly employed in 3DGS-based SLAM systems. As shown in Fig. 2 , our method significantly enhances both the reconstruction accuracy and rendering quality while maintaining the global consistency of the semantic map. (4) Extensive experiments conducted on Replica [21] and Scan-Net [2] datasets demonstrate that our GS 3 LAM outperforms its counterparts in terms of tracking accuracy, rendering quality and speed, and semantic reconstruction. 
related work 2.1 scene representation for semantic slam: 
 Semantic SLAM systems typically utilize various scene representations such as points/surfels [15, 23, 26, 28] , grids [16] , or voxels [8, 11, 17] to facilitate the creation of semantic maps. For instance, the mesh-based SLAM++ [18] models the world as a graph, with each node capturing an estimated 𝑆𝐸 (3) pose, and represents each 3D object as a mesh. Another notable system, Kimera [1] , annotates semantic labels onto the faces of meshes, enabling the real-time construction of metric-semantic 3D mesh environment models.The surfel-based SemanticFusion [13] , on the other hand, builds upon real-time ElasticFusion [28] and utilizes CNN predictions of pixel categories and Bayesian update schemes to track the category probability distribution of each surfel, thereby establishing a globally consistent semantic map. Despite the benefits that these representation techniques present in terms of geometry, storage, computational efficiency, and scalability, they encounter difficulties in predicting unexplored areas and are restricted by limited resolutions, thereby incapable of producing dense semantic maps.In recent years, neural rendering techniques based on continuous scene representations, such as Neural Radiance Fields (NeRF) [14] and 3D Gaussian Splatting (3DGS) [7] , have emerged, showing significant potential in photorealistic rendering and dense reconstruction. NeRF represents scenes as continuous implicit volume functions, enabling realistic novel view synthesis with modest storage requirements. For NeRF-based SLAM, existing methods can be categorized into two main types, implicit (MLP-based) representation methods and hybrid representation methods. The MLP-based iMAP [24] is the first to employ neural radiance for tracking and mapping tasks, offering memory-efficient dense map representations but failing to scale to large scenes. On the other hand, hybrid representation methods combine the scalability of explicit representations with the low memory consumption of implicit representations, significantly improving scene scalability and accuracy. For instance, NICE-SLAM [34] proposes hierarchical multi-feature grids, Co-SLAM [25] adopts multi-resolution hash grids, and Vox-Fusion [30] utilizes octrees for dynamic map expansion. ESLAM [5] and Point-SLAM [19] , in addition, employ tri-planes and neural point clouds respectively for volume rendering, significantly enhancing mapping capabilities. Furthermore, some methods [9, 32] incorporate additional MLP channels to encode and decode semantic labels, while optimizing camera poses and semantic scenes simultaneously. However, due to the computational expense of NeRF's ray-tracingbased volume rendering, these methods fail to meet the real-time requirements of SLAM.In contrast to NeRF, 3DGS achieves remarkable capabilities by representing scenes as dense Gaussian clouds and a tile-based rasterization, thereby accomplishing high-quality and efficient rendering. Recently, several SLAM methods [4, 6, 12, 29] based on 3DGS have been developed. They represent scenes as 3D Gaussians and directly backpropagate to optimize camera poses and the Gaussian fields.Notably, the aforementioned NeRF-based and 3DGS-based SLAM predominantly focus on the optimization of RGB maps, which poses limitations on their direct applicability to downstream tasks such as navigation and localization. To address this challenge, we propose an approach that integrates semantic features into 3D Gaussians and jointly optimizes camera poses, geometry, appearance, and semantics, enabling accurate 3D semantic mapping, robust camera tracking, and high-quality real-time rendering. 
methodology 3.1 framework overview: 
 As illustrated in Fig. 3 , our GS 3 LAM framework is designed to process RGB-D data with unknown camera poses and corresponding 2D semantic labels. It models the scene as a SG-Field, wherein each 3D Gaussian is characterized by its position µ, rotation matrix R, scaling matrix S, opacity 𝑜, color c, and semantic feature f. To facilitate progressive reconstruction of semantic maps with geometric-semantic consistency, we employ an adaptive 3D Gaussian expansion technique and propose the RSKM strategy to alleviate the forgetting phenomenon. Finally, GS 3 LAM optimizes camera poses and the SG-Field using appearance, geometry, and semantics, along with the proposed DSR scheme which ensures the alignment between geometry and semantics within the field.Our goal is to establish a scene representation that efficiently captures the geometry, appearance, and semantics of the scene, thereby facilitating the production of realistic dense map and precise semantic reconstruction. To accomplish this objective, we model the scene as a SG-Field G containing 𝑁 semantic Gaussians,EQUATIONwhere the 𝑖-th 3D semantic Gaussian is defined by its position µ 𝑖 ∈ R 3 , covariance matrix 𝚺 𝑖 ∈ R 3×3 , opacity 𝑜 𝑖 ∈ R, RGB color c 𝑖 ∈ R 3 , and semantic feature f 𝑖 ∈ R 𝑁 𝑠𝑒𝑚 (𝑁 𝑠𝑒𝑚 denotes the number of objects in the field). To optimize the parameters of the SG-Field using gradient descent, the covariance matrix Σ 𝑖 can be represented equivalently as [7] ,EQUATIONwhere S 𝑖 ∈ R 3×3 represents a diagonal scaling matrix, and R 𝑖 ∈ R 3×3 denotes a rotation matrix.Color and Depth Splatting-Rendering. When provided with an optimized SG-Field G, along with a world-to-camera viewing transformation (also known as the camera pose) T 𝐶𝑊 ∈ R 4×4 , the 𝑖-th 3D semantic Gaussian can be projected onto the 2D image plane for rendering with a 2 × 2 covariance matrix Σ 2𝐷 𝑖 [35] ,EQUATIONwhere J 𝑖 ∈ R 2×3 is the Jacobian of the 𝑖-th Gaussian centroid projected onto the 2D image plane with respect to its position in the camera coordinate system, and R 𝐶𝑊 ∈ R 3×3 denotes the rotation matrix of the camera pose T 𝐶𝑊 . Upon the projection of 3D Gaussians onto the image plane, the color of a single pixel ĉ𝑝𝑖𝑥 is rendered by sorting the Gaussians in depth order and performing front-to-back 𝛼-blending rendering as, where 𝑀 is the number of sorted Gaussians overlapping with the given pixel. The density 𝛼 𝑖 is computed from the 2D covariance matrix Σ 2𝐷 𝑖 and the opacity 𝑜 𝑖 of the 𝑖-th 3D Gaussian as,EQUATIONEQUATIONwhere σ 𝑖 ∈ R 2 is the offset between the pixel center and the 𝑖-th projected 2D Gaussian center. Likewise, the depth d𝑝𝑖𝑥 of a single pixel is rendered by,EQUATIONwhere 𝑑 𝑖 is the depth of the 𝑖-th Gaussian centroid with respect to the camera coordinate system.To develop a versatile pipeline for embedding semantic features, it is imperative that our approach possesses the capability to generate semantic feature maps of varying sizes and dimensions. To fulfill this requirement, we employ a rendering pipeline based on differentiable 3DGS framework similar to color and depth. Specifically, the 2D semantic feature of a single pixel f𝑝𝑖𝑥 can be rendered as,EQUATIONwhere f 𝑖 denotes the 𝑁 𝑠𝑒𝑚 -dimensional semantic feature vector of the 𝑖-th 3D Gaussian. To decode discrete semantic labels from continuous 2D semantic features, we initially utilize a CNN decoder F 𝑐𝑛𝑛 to restore the low-dimensional feature to 𝐾 𝑠𝑒𝑚 dimensions (𝐾 𝑠𝑒𝑚 represents the semantic label categories). Then, a softmax classification is employed on the high-dimensional feature to obtain the semantic label ŝ𝑝𝑖𝑥 of a single pixel,EQUATIONDue to 𝑁 𝑠𝑒𝑚 ≪ 𝐾 𝑠𝑒𝑚 , GS 3 LAM can efficiently achieve the conversion between 3D semantic features and 2D semantic labels, seamlessly embedding semantic features into 3DGS-based SLAM while maintaining the optimization efficiency.In our GS 3 LAM system, the parameters to be optimized include 𝑃 camera poses T and the SG-Field G,EQUATIONwhere q 𝑖 = [𝑞 𝑤 𝑖 , 𝑞 𝑥 𝑖 , 𝑞 𝑦 𝑖 , 𝑞 𝑧 𝑖 ] 𝑇 represents the rotation quaternion, t 𝑖 = [𝑡 𝑥 𝑖 , 𝑡 𝑦 𝑖 , 𝑡 𝑧 𝑖 ] 𝑇 denotes the translation vector, and the parameters of the SG-Field G are defined in Eq. (1) and Eq. (8) . Simultaneously optimizing both the camera pose parameters 𝚯 T and the semantic Gaussian parameters of SG-Field 𝚯 G is time-consuming and challenging. Therefore, a strategy of decoupling the optimization of camera poses and field parameters is adopted. In the tracking stage (Sec. 3.4), GS 3 LAM optimizes the camera pose of the current frame T 𝑡 with reference to a pre-trained SG-Field G 𝑡 -1 . During the mapping phase (Sec. 3.3), it optimizes the current SG-Field G 𝑡 based on accurately estimated camera poses T 0 , T 1 , . . . , T 𝑡 .3.3.1 Adaptive 3D Gaussian Expansion. To accommodate to the paradigm of incremental reconstruction in SLAM, an adaptive 3D Gaussian expansion strategy is employed during the mapping process. Following the tracking of a frame, we re-render the current frame and compute the cumulative opacity ô𝑝𝑖𝑥 for each pixel. This process can be seamlessly integrated into the differentiable rasterization pipeline of 3DGS [7] ,EQUATION)Inspired by [6, 29] , cumulative opacity and depth are employed to construct a mask for the unobservable regions of the SG-Field G 𝑡 -1 under the viewpoint T 𝑡 in the current frame,EQUATION)where I denotes the indicator function, 𝜏 𝑢𝑛𝑜𝑏𝑠 represents cumulative opacity threshold for unobservable regions, and L1 (𝑑 𝑔𝑡 , d𝑝𝑖𝑥 ) refers to the median of the 𝑙 1 -norm error between the observed depth 𝑑 𝑔𝑡 and the rendered depth d𝑝𝑖𝑥 . This mask indicates regions characterized by inadequate map density ( ô𝑝𝑖𝑥 < 𝜏 𝑢𝑛𝑜𝑏𝑠 ), or where additional geometry is anticipated to exist ahead of the presently estimated geometry (𝐿 1 (𝑑 𝑔𝑡 , d𝑝𝑖𝑥 ) > 50 L1 (𝑑 𝑔𝑡 , d𝑝𝑖𝑥 )). Relying on this mask, we can dynamically and adaptively integrate newly observed regions into the SG-Field (G 𝑡 -1𝑀 unobs -----→ G 𝑡 ).Concurrently, this mask serves to prevent the addition of new Gaussians to areas where the current Gaussian adequately represents the scene geometry, thereby effectively managing the number of Gaussians within G 𝑡 , leading to decreased memory usage and optimization time.Based on the mask 𝑀 𝑢𝑛𝑜𝑏𝑠 of the current frame, all unobservable pixels are used to expand new semantic Gaussians. Specifically, for each pixel, we add a new semantic Gaussian with the color of that pixel, the semantic feature represented by random 𝑁 𝑠𝑒𝑚 -dimensional Spherical Harmonics coefficients, the centroid at the location of the unprojection of that pixel depth 𝑑 𝑔𝑡 , an opacity of 0.5, and scales initialized to 𝑑 𝑔𝑡 /𝑓 , where 𝑓 denotes the camera focal length. Although this scaling initialization strategy shows higher efficiency compared to the KNN method in 3DGS [7] , the variations in depth range across different frames result in significant variance in the 3D Gaussian scales corresponding to these frames. Such variance is not conducive to the SG-Field optimization. Furthermore, this strategy fails to adaptively represent high-and low-frequency information within the field, i.e., using smaller scales in high-frequency regions and larger scales in low-frequency regions. To address these challenges, we propose a depth-adaptive scale regularization term,EQUATIONwhere 𝑠 𝑖 denotes the scale of the 𝑖-th Gaussian, 𝑠 𝑏𝑖𝑔 and 𝑠 𝑠𝑚𝑎𝑙𝑙 adhere the 2𝜎 rule, i.e., 𝑠 𝑏𝑖𝑔 = 𝜇 𝑠 + 2𝜎 𝑠 and 𝑠 𝑠𝑚𝑎𝑙𝑙 = 𝜇 𝑠 -2𝜎 𝑠 . These terms constrain the global Gaussian scales within a reasonable range (𝜇 𝑠 -2𝜎 𝑠 < 𝑠 < 𝜇 𝑠 +2𝜎 𝑠 ), thereby preventing excessively large or small Gaussians. Moreover, they indirectly align the Gaussians with geometric surfaces, reducing the blurriness of object edges, and achieving spatial alignment between geometry and semantics.As illustrated in Fig. 4 , due to the optimization properties of 3DGS [7] , 3DGS-based SLAM systems inherently demonstrate a propensity for forgetting during the incremental reconstruction procedure.In order to alleviate this issue, SplaTAM [6] and MonoGS [12] adopt a strategy of Local Co-visible Keyframe Mapping (LCKM), wherein, during the optimization of the current frame, the remaining keyframes co-visible with the current frame are selected to participate in optimization together. However, as shown in Fig. 2 , we observe that this approach leads to under-optimized regions with sparse co-visibility, while areas with numerous co-visibility frames exhibit a tendency towards convergence difficulties, resulting in significantly biased semantic maps. To address this problem, we propose the RSKM strategy, which effectively reduces the optimization bias and enhances the global consistency of the SG-Field.In the process of mapping the current frame 𝑓 𝑐𝑢𝑟 , during each iteration, RSKM selects a frame 𝑓 in the keyframe set K with probability 𝑝 (𝑓 ) to participate in the optimization,EQUATIONwhere |K | denotes the size of the keyframe set K, 𝑘 𝑚 represents the number of iterations for mapping, 𝛿 𝑖,𝑗 is the Kronecker delta function, which equals 1 if 𝑖 = 𝑗 and 0 otherwise, and the optimization target interval 𝑡 𝑜𝑝𝑡 is used to balance optimization between the current frame and keyframes. It is noteworthy that RSKM does not involve time-consuming keyframe selection operations as done in SplaTAM [6] and Point-SLAM [19] , yet still achieves a high level of effectiveness in ensuring the global consistency of semantic maps.Based on the aforementioned sampling strategy, the optimization objective of SG-Field G 𝑡 can be defined as,EQUATIONwhere 𝑘 𝑚 represents the number of iterations for mapping, L 𝑚𝑎𝑝𝑝𝑖𝑛𝑔 refers to the mapping loss, T 𝑖 and O 𝑖 represent the camera pose and the ground truth data (RGB image, depth map and semantic labels) of the associated frame respectively, R denotes rasterization rendering, and ⊙ represents the transformation of G 𝑡 with T 𝑖 .To ensure the consistency of multimodality within G 𝑡 , L 𝑚𝑎𝑝𝑝𝑖𝑛𝑔 encompasses constraints related to appearance, semantics, geometry, and geometric-semantic spatial alignment. The color loss L 𝑚 𝑐𝑜𝑙𝑜𝑟 is an 𝑙 1 loss combined with a D-SSIM [27] term,EQUATION)where ĉ𝑝𝑖𝑥 and c 𝑔𝑡 denote the rendered and observed color, and we use 𝜆 = 0.2 in all our tests. A binary cross entropy (BCE) loss is applied as the semantic loss,EQUATION)where ŝ𝑝𝑖𝑥 is the decoded semantic label from the semantic feature, and 𝑠 𝑔𝑡 is the input semantic label provided by the dataset or generated using state-of-the-art semantic segmentation models. An 𝑙 1 depth loss is utilized to guide geometry,EQUATIONwhere d𝑝𝑖𝑥 and 𝑑 𝑔𝑡 are rendered depth and ground truth depth.Finally, the inclusion of the regularization terms for the Gaussian scales from Eq. ( 12) constitutes the complete mapping loss L 𝑚𝑎𝑝𝑝𝑖𝑛𝑔 ,EQUATIONwhere 𝜆 𝑚 𝑐 , 𝜆 𝑚 𝑑 , 𝜆 𝑚 𝑠 , 𝜆 𝑚 𝑏𝑖𝑔 , and 𝜆 𝑚 𝑠𝑚𝑎𝑙𝑙 control the weight of each term.Given an optimized SG-Field G 𝑡 -1 , GS 3 LAM employs the frameto-model strategy to optimize the world-to-camera poses T . In particular, for the first frame, the camera pose T 0 is initialized as the identity matrix. Then, adhering to the methodology outlined in Sec.3.3, all pixels are initialized as Gaussians, and the mapping process is executed for 𝑘 𝑖𝑛𝑖𝑡 iterations to yield the initially optimized G 0 .When a new frame arrives, GS 3 LAM initializes the camera pose T 𝑡 using the constant velocity assumption as,EQUATIONThen, the SG-Field G 𝑡 -1 is transformed into the camera coordinate system via T 𝑡 , which is optimized by minimizing the tracking loss L 𝑡𝑟𝑎𝑐𝑘𝑖𝑛𝑔 between the rendered R (•) and the ground truth data O,EQUATION)It is noteworthy that during the aforementioned optimization process, all attributes of the SG-Field G 𝑡 -1 are frozen, separating the camera movement from the deformation, densification, pruning, and self-rotation of the 3D Gaussian points. It is apparent that the SG-Field G 𝑡 -1 inadequately observes all regions within the current frame. To improve the robustness and stability of tracking, L 𝑡𝑟𝑎𝑐𝑘𝑖𝑛𝑔 is designed to be aware of observable and geometrically normal regions, jointly minimizing photometric, geometric, and semantic errors,EQUATIONwhere 𝑀 𝑜𝑏𝑠 denotes the mask of well-optimized depth in observable regions ( ô𝑝𝑖𝑥 > 𝜏 𝑜𝑏𝑠 ) of the SG-Field G 𝑡 -1 under the viewpoint T 𝑡 , which holds significant importance for tracking. L 𝑡 𝑐𝑜𝑙𝑜𝑟 = ĉ𝑝𝑖𝑥c 𝑔𝑡 1 solely employs the 𝑙 1 loss, and 𝜆 𝑡 𝑐 , 𝜆 𝑡 𝑑 , 𝜆 𝑡 𝑠 modulate the weight of each term. 
experiment 4.1 setup: 
 4.1.1 Implementation Details. GS 3 LAM was implemented in Python using the PyTorch framework, and trained on a workstation with an AMD EPYC 7302 16-Core Processor and an NVIDIA GeForce RTX 3090 GPU. For SG-Field, we employed isotropic Gaussians, with semantic feature dimension 𝑁 𝑠𝑒𝑚 = 16 and semantic label categories 𝐾 𝑠𝑒𝑚 = 256. Regarding the cumulative opacity mask, we set 𝜏 𝑢𝑛𝑜𝑏𝑠 = 0.5 for adaptive 3D Gaussian expansion and 𝜏 𝑜𝑏𝑠 = 0.99 for observable region-aware tracking. As for RSKM, the optimization target interval 𝑡 𝑜𝑝𝑡 was set to 10 for mapping. For the initial frame of mapping, the number of iterations 𝑘 𝑖𝑛𝑖𝑡 was set to 1000 for Replica [21] and 500 for Scannet [2] .4.1.2 Datasets and Evaluation Metrics. Following [6, 19, 30, 33, 34] , we used 8 scenes from the virtual Replica [21] and 5 subsets of realworld ScanNet [2] for tracking and rendering quality comparison. Rendering quality was assessed utilizing objective metrics including Peak Signal-to-Noise Ratio (PSNR), SSIM [27] , and LPIPS [31] . Tracking accuracy was quantified by the ATE RMSE [22] . Semantic segmentation performance was gauged using the mean Intersection over Union (mIoU). In all of our tables, best results are highlighted as first and second .We conducted a comparative analysis between our proposed GS 3 LAM and several state-of-the-art dense neural RGBD SLAM methodologies, including NICE-SLAM [34] , Vox-Fusion [30] , ESLAM [5] , Co-SLAM [25] and Point-SLAM [19] . Additionally, we expanded our comparison to encompass leading 3DGS-based SLAM techniques, specifically SplaTAM [6] and GS-SLAM [29] . In the context of semantic reconstruction, our method underwent evaluation against NeRF-based NIDS SLAM [3] , DNS SLAM [9] and SNI-SLAM [32] .Tables 1 and 2 present the comparative rendering results of GS 3 LAM with state-of-the-art NeRF-based and 3DGS-based SLAM systems on the ScanNet [2] and Replica [21] datasets, respectively. The results demonstrate that GS 3 LAM achieves the best performance across commonly used metrics. On the Replica dataset, our approach outperforms the runner-up methods Point-SLAM [19] and GS-SLAM [29] by 1.09 dB, 0.014, and 0.039 in terms of PSNR, SSIM and LPIPS, respectively. Moreover, on the real-world ScanNet dataset, our superiority is more pronounced, with our method surpassing Point-SLAM [19] by 3.04 dB in PSNR, 0.117 in SSIM, and 0.292 in LPIPS. Compared to the 3DGS-based SplaTAM [6] and GS-SLAM [29] , the semantic embedding and DSR scheme in GS 3 LAM enable the Gaussian model to focus more on the details of object edges and eliminate surface blurring. Additionally, our proposed RSKM strategy effectively addresses the challenge of convergence in regions with abundant covisibility, as well as the issue of suboptimal optimization in regions with sparse covisibility, achieving a balance between local and global optimization. This approach effectively alleviates the forgetting phenomenon inherent in 3DGSbased SLAM, thereby facilitating globally consistent and realistic rendering performance. Qualitatively, the visualization results in Fig. 5 demonstrate that NeRF-based Co-SLAM [25] and Point-SLAM Figure 5 : Qualitative comparison with SOTA methods on virtual Replica [21] and real-world ScanNet [2] datasets.[19] exhibit inaccurate scene representations and are susceptible to lighting effects, leading to significant artifacts. SplaTAM [6] tends to get trapped in local optima, making convergence difficult or suboptimal, resulting in noticeable holes and blurring. In contrast, GS 3 LAM produces higher-quality and more realistic images with more structure details in both global and edge regions compared to other methods. It is noteworthy that, owing to the efficient semantic scene representation of SG-Field and the tile-based rasterization technology, GS 3 LAM achieves real-time rendering of RGB, depth, and semantics at 109.12 FPS on the 1200 × 680 Replica dataset, a 36.86-fold improvement over NeRF-based SLAM methods. Similarly, on the 640 × 480 ScanNet dataset, it reaches 499.78 FPS, providing possibilities for downstream real-time tasks. More results can be found in the supplemental materials. Table 3 presents a comparison of the tracking performance between GS 3 LAM and state-of-the-art NeRF-based and 3DGS-based SLAM systems on the Replica [21] dataset. Since these methods employ a frame-to-model tracking strategy, SG-Field can more accurately represent the scene compared to NeRF-based methods, thus resulting in higher tracking precision. In contrast to SplaTAM [6] , although semantic embedding allows GS 3 LAM to focus more on the edges and details of the field, tracking relies on prominent features rather than details, thereby leading to a slight decrease in accuracy.Table 4 presents the quantitative comparison between GS 3 LAM and several contemporary neural semantic SLAM approaches. Following the protocol outlined in NIDS-SLAM [3] , we report the mean Intersection over Union (mIoU) across four scenes from the Replica dataset [21] . From Table 4 , it can be observed that leveraging SG-Field for semantic feature embedding within GS 3 LAM leads to noticeable enhancements (increased by 9.22%) when compared with competing NeRF-based semantic methods. Cumulative Opacity Mask Ablation. As evidenced by the ablation experiments in Table 5 , the utilization of cumulative opacity masks is pivotal within the GS 3 LAM framework. During the tracking stage, the observable region mask 𝑀 𝑜𝑏𝑠 serves as the basis for decoupling camera pose estimation and SG-field optimization. It prevents the influence of unoptimized SG-Field on the current frame's tracking, reducing tracking errors from 43.12cm to 0.21cm. In mapping, compared to randomly sampling pixels from the current frame for expansion, the unobserved region mask 𝑀 𝑢𝑛𝑜𝑏𝑠 filters regions already optimized, thereby avoiding the addition of new Gaussians to regions already represented by Gaussians. This effectively controls the number of Gaussians in the field and improves PSNR by 2.22 dB and mIoU by 6.91%.DSR Ablation. As depicted in Fig. 6 , the absence of DSR strategy results in the emergence of numerous Gaussians with large scales at scene edges or unobserved regions, leading to blurriness at object boundaries and spatial misalignment between geometry and semantics. Furthermore, as shown in Table 5 , the clear geometric contours achieved by DSR can reduce tracking errors by 16%, increase PSNR by 1.17 dB, and enhance semantic reconstruction accuracy by 3.36%.RSKM Ablation. As illustrated in Fig. 7 , our proposed RSKM achieves a 5.49 dB increase in PSNR while reducing the variance by 24.42 times, mitigating the optimization bias of the SG-Field and ensuring consistency in rendering across all perspectives. When RSKM is not employed (using LCKM instead), in regions with a high number of co-observed frames, the SG-Field undergoes repetitive optimization across frames, making it challenging to converge. Conversely, in regions with fewer co-observed frames, under-optimization occurs due to insufficient sampling. Consequently, LCKM results in numerous holes and blurriness in the field. Furthermore, as indicated in Table 5 , RSKM also reduces tracking errors and enhances semantic reconstruction accuracy, contributing tremendously to achieving a globally consistent map in terms of geometry, semantics, and appearance.  
conclusion: 
 We propose GS 3 LAM, a Gaussian Semantic Splatting SLAM system that utilizes 3D semantic Gaussians for dense map construction and tracking. Leveraging semantic Gaussian field scene representation, our approach better captures appearance, geometry, and semantics within the scene. Additionally, our proposed depth-adaptive scale regularization strategy adaptively adjusts the scales of Gaussians to characterize the scene, reducing uncertainties of 3D Gaussians at object surfaces and edges, thereby enhancing the accuracy of the 3D scene representation and achieving spatial alignment between geometry and semantics. Moreover, our proposed simple yet powerful random sampling-based keyframe mapping strategy effectively reduces optimization biases, mitigates the exacerbation of the forgetting phenomenon induced by semantic feature embedding, and enhances the global consistency of the semantic map. Thorough evaluations on benchmark datasets corroborate that GS 3 LAM outperforms its rivals noticeably in terms of tracking accuracy, rendering quality and speed, and semantic reconstruction.  
: 
 a Gaussian Semantic Splatting SLAM framework, which takes multimodal data as input and can render consistent, continuous dense semantic maps in real-time. To fuse multimodal data, GS 3 LAM models the scene as a Semantic Gaussian Field (SG-Field), and jointly optimizes camera poses and the field by establishing error constraints between observed and predicted data. Furthermore, a Depth-adaptive Scale Regularization (DSR) scheme is proposed to tackle the problem of misalignment between scale-invariant Gaussians and geometric surfaces within the SG-Field. To mitigate the forgetting phenomenon, we propose an effective Random Samplingbased Keyframe Mapping (RSKM) strategy, which exhibits notable superiority over local covisibility optimization strategies commonly utilized in 3DGS-based SLAM systems. Extensive experiments conducted on the benchmark datasets reveal that compared with stateof-the-art competitors, GS 3 LAM demonstrates increased tracking robustness, superior real-time rendering quality, and enhanced semantic reconstruction precision. To make the results reproducible, the source code will be publicly released.• Computing methodologies → Reconstruction.Semantic SLAM, Gaussian splatting, 3D segmentation 
 
 Appendix:  


Decision: The paper receives mixed ratings from three reviewers. Two reviewers recommended positive recommendations. They acknowledged the technical novelty and the performance of the method. The second reviewer first gave a borderline accept and then changed the rating to borderline reject after the rebuttal. The major concern is that the rebuttal does not answer the question related to overall performance comparison to the competing methods, including both accuracy and efficiency. AC double-checked the experimental section and found that the comparison to the existing GS-based and NeRF-based methods shows clear improvements. Considering the positive majority of comments, AC finally recommends acceptance of this submission.

Reviews: review: paper_summary: paper_summary: This paper introduces a new SLAM (Simultaneous Localization and Mapping) framework that combines semantic understanding with 3D Gaussian splatting for better tracking, high-quality rendering, and accurate semantic reconstruction. It uses Semantic Gaussian Fields (SG-Fields) and improves performance with depth-adaptive scale regularization (DSR) and random sampling-based keyframe mapping (RSKM).
strength_weakness: strength_weakness: The integration of semantic understanding into SLAM using 3D Gaussian splatting effectively captures appearance, geometry, and semantics.

The proposed DSR scheme improves alignment between geometry and semantics, reducing blurring and enhancing accuracy.

The proposed RSKM address the optimization bias and the forgetting phenomenon. Visual results show that RSKM enhances the consistency and accuracy of the semantic map, reducing artifacts and blurriness.

The extensive experiments conducted reveal that GS3LAM outperforms state-of-the-art methods in terms of tracking robustness, rendering quality, and semantic reconstruction precision.
limitations: limitations: The paper does not discuss other recent published works on Gaussian Splatting SLAM, such as [1], which may offer relevant insights and benchmarks.

The paper introduces the problem of forgetting early in the introduction but delays its definition and explanation until Method 3 and Figure 4. For better comprehension, the problem should be defined and explained earlier in the paper.

While the paper conducts extensive experiments on the ScanNet and Replica datasets, it lacks evaluations on additional datasets. The referenced works [1], [2], [3], [4], and [5] have included experiments on at least one or two other datasets, such as TUM-RGBD, in addition to ScanNet and Replica. This makes the experimental validation in this paper appear somewhat limited. To enhance the robustness and comprehensiveness of the evaluation, it is recommended to include results on the TUM-RGBD dataset.

Reference:

[1] Gaussian Splatting SLAM (CVPR 2024)

[2] SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM

[3] Point-SLAM: Dense Neural Point Cloud-based SLAM

[4] Vox-Fusion: Dense Tracking and Mapping with Voxel-based Neural Implicit Representation

[5] NICE-SLAM: Neural Implicit Scalable Encoding for SLAM
paper_summary: paper_summary: This paper introduces a novel semantic gaussian splatting SLAM system named GS3LAM. The system utilizes 3D semantic gaussian representation and a differentiable splatting rasterization pipeline to jointly optimize camera poses and fields for appearance, geometry, and semantics, achieving robust tracking, real-time high-quality rendering, and precise 3D semantic reconstruction.
strength_weakness: strength_weakness: - The paper introduces a new approach to SLAM by utilizing a 3D semantic Gaussian representation combined with a differentiable splatting rasterization pipeline. By integrating RGB, depth, and semantic information, this proposed GS3LAM can consistent and continuous dense semantic maps in real-time.
- The “Depth-adaptive Scale Regularization (DSR) scheme” and “Random Sampling based Keyframe Mapping (RSKM) strategy” help the SLAM system to better represent the scene and mitigate the forgetting phenomenon.
- Comprehensive evaluations have demonstrated GS3LAM's clear advantages over competing systems. The ablation analysis also illustrates the effectiveness of each component of the SLAM system.
limitations: limitations: Comments on the Methods:
- Some methodological formulas are not proposed by the author, such as Eq.19 and Eq.21. The relevant references should be added.
- The expressions in some Equations are irregular. What does the formulars “II” mean in Eq. (12)? 
Comments on the Experiments:
- Lack of tracking comparison with semantic NeRF/GS-based SLAM methods such as NIDS SLAM, DNS SLAM and SNI-SLAM.
- In the comparison experiment of Rendering and Tracking, the superiority of this method is proved, but the comparison of Reconstruction results is missing.
- The Table4 in the Supplementary Materials shows the tracking results of the in the real-world dataset, ScanNet. Obviously, the localization accuracy of the proposed method in the real scene is lower than the others. The explanation should be provided.
- Runtime Analysis is a very important indicator in the SLAM system. According to the comparison results in Table1 of Supplementary Materials, GS3LAM has no advantage in the overall running efficiency, only the rendering speed is faster.
- It is recommended that the author give a comparison of Memory Usage parameters.
Other comments:
- Miswriting "known" as "konwn" in Abstract.
- PSNR in Figure1 should be the result of RGB rendering, but “Ours” is a semantic image.
paper_summary: paper_summary: In this paper, authors propose a Gaussian Splatting Semantic SLAM framework(GS3LAM) including two main modules. The first Depth-adaptive Scale Regularization (DSR) module constrains scales within a depth-dependent range to align Gaussians with geometric surfaces. The second Random Sampling-based Keyframe Mapping (RSKM) module is presented to address the forgetting phenomenon in GS3LAM. Extensive experiments conducted on Replica and ScanNet datasets demonstrate that our GS3LAM outperforms its counterparts in terms of tracking accuracy, rendering quality and speed, and semantic reconstruction.
strength_weakness: strength_weakness: Extend 3DGS as a semantic scene representation is interesting and urgent research topic.
Overall, the motivations for the two main innovations proposed by the authors are sufficient and reasonable.
limitations: limitations: The experimental section still needs improvement and refinement. 1、The author compares the open-source Gaussian schemes, but can add some works that have already been published in journals. 2、most important, In terms of algorithm superiority, the best and overall performance do not surpass the two Gaussian open-source works being compared. 3 The author claims that the Gaussian-based scheme is superior in time performance compared to the NeRF-based scheme, but this is not reflected in the experimental section.
paper_summary: paper_summary: Traditional semantic SLAM methods using explicit representations are limited in resolution and predictive capabilities. Inspired by the Gaussian Splatting, this paper introduced GS3LAM, a framework that inputs multimodal data to produce real-time, consistent, dense semantic maps. It features a Semantic Gaussian Field for scene modeling, Depth-adaptive Scale Regularization for alignment accuracy, and Random Sampling-based Keyframe Mapping for optimization. Extensive testing shows GS3LAM leads in tracking, rendering, and semantic reconstruction.
strength_weakness: strength_weakness: 1. Method Innovation: This paper presents a pioneering approach to SLAM through the innovative use of a 3D semantic Gaussian representation. Furthermore, the paper introduces two key enhancements to the SLAM system: the "Depth-adaptive Scale Regularization (DSR) scheme" and the "Random Sampling based Keyframe Mapping (RSKM) strategy" to enhance the precision of the 3D scene representation.
2. Experiments Analysis: The paper has undergone a thorough evaluation process, which has conclusively demonstrated the superior performance of GS3LAM in comparison to other SOTA systems. The systematic ablation analysis further delves into the individual components of the SLAM framework, meticulously highlighting the contribution and effectiveness of each element.
limitations: limitations: 1. Expanded Tracking Comparison: While the paper has established the method's superiority in rendering and tracking, a comparative analysis of tracking performance with other semantic NeRF/GS-based SLAM methods like NIDS SLAM, DNS SLAM, and SNI-SLAM would be beneficial. This would offer a more holistic view of the method's capabilities and standing within the field.
2. Localization Accuracy in Real-world Scenarios: The tracking results presented in Table 4 of the Supplementary Materials indicate a lower localization accuracy for the proposed method in real-world datasets like ScanNet. The authors should explain this performance to offer context and potential reasons for the observed results.
3. Memory Usage Comparison: Memory usage is a vital parameter, especially for systems that may be deployed on devices with limited resources. It is recommended that the authors include a comparison of memory usage to provide a complete picture of the system's resource requirements and efficiency.
4. Clarification on Mathematical Expressions: The notation "II" in Equation (12) appears to be non-standard and may be a typographical error. For precise understanding and application of the methodology, it would be essential for the authors to provide a clear definition or correct any typos to ensure the equations are interpreted correctly.