SAR-SLAM: Self-Attentive Rendering-based SLAM with Neural Point Cloud Encoding

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 10:58:32

Abstract: Neural implicit representations have recently revolutionized simultaneous localization and mapping (SLAM), giving rise to a groundbreaking paradigm known as NeRF-based SLAM. However, existing methods often fall short in accurately estimating poses and reconstructing scenes. This limitation largely stems from their reliance on volume rendering techniques, which oversimplify the modeling process. In this paper, we introduce a novel neural implicit SLAM system designed to address these shortcomings. Our approach reconstructs Neural Radiance Fields (NeRFs) using a self-attentive architecture and represents scenes through neural point cloud encoding. Unlike previous NeRF-based SLAM methods, which depend on traditional volume rendering equations for scene representation and view synthesis, our method employs a self-attentive rendering framework with the Transformer architecture during mapping and tracking stages. To enable incremental mapping, we anchor scene features within a neural point cloud, striking a balance between estimation accuracy and computational cost. Experimental results across three challenging datasets demonstrate the superior performance and robustness of our proposed approach compared to recent NeRF-based SLAM systems. The code will be released.

Main paper: 
 introduction: 
 Simultaneous Localization and Mapping (SLAM) [4] stands as a fundamental challenge in computer vision, finding wide-ranging applications in autonomous driving [3] , robot navigation [5] , augmented reality (AR), virtual reality (VR), collision detection [7] , and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym 'XX, June 03-05, 2018, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/18/06 https://doi.org/XXXXXXX.XXXXXXX scene perception [27] . With the escalating demand for high-fidelity 3D scene reconstruction, there's a critical shift towards representing target scenes more accurately, favoring continuous surfaces over discrete point clouds. Despite the remarkable strides made in 3D scene reconstruction technology in recent years, achieving realtime representation of high-quality scenes without compromising accuracy and spatial resolution remains a persistent challenge.The advent of low-cost visual sensors has led to the emergence of numerous real-time dense visual SLAM systems, gaining substantial attention across academia and industry. Traditional methods in dense visual SLAM employ representations such as point clouds, surfels, voxel grids, voxel hashing, or octrees to achieve real-time, dense, and large-scale scene reconstruction [30, 47] . However, these methods often struggle to provide accurate geometric estimations for unobserved areas. In contrast, learning-based SLAM methods exhibit predictive capabilities as they are typically trained on taskspecific datasets [1, 16, 38, 43, 48 ]. Yet, many of these methods rely on ground truth depth or 3D mesh for training, posing challenges in generalization to unseen scenes during testing. Furthermore, deep learning-based scene representation encounters challenges with local scene updating and fixed network capacity during runtime, constraining their capability for achieving high-fidelity reconstruction in large scenes.With the introduction of Neural Radiance Fields (NeRF) [20] and its diverse applications in inverse rendering [32] , controllable editing [23] , digital human body generation [40] , multi-modality [37] , image and video processing [15] , medical imaging [31] , and various other fields, researchers have begun integrating NeRF into SLAM research. The implicit representation produced by the continuous radiance field enables high-quality rendering of both visible and occluded regions, facilitating the extraction of underlying shapes at any resolution. Compared to conventional methods, implicit neural scene representations demonstrate superior noise and outlier suppression capabilities in mapping, enhanced hole-filling and inpainting capacities for occluded scenes, and more robust data compression abilities. Despite showcasing significant performance advancements, these approaches often involve decoding the feature volume into a radiance field and rely on classic volume rendering formulas for view synthesis. It's important to note that the volume rendering equation utilized in NeRF oversimplifies optical modeling concerning solid surfaces [45] , reflectivity [6, 8] , and inter-surface scattering effects. Consequently, within NeRF, the brightness field associated with volume rendering does not present a universal imaging model, thereby limiting the generalizability of NeRF-based SLAM algorithms when encountering new scenes.Drawing inspiration from IBRNet [39] , we present SAR-SLAM, a NeRF Transformer-based SLAM system with neural point cloud encoding designed for improving generalizability. As illustrated in Fig. 1 , our approach hinges on several key ideas. 1) Instead of relying on classical volume rendering equations for view synthesis, we employ a ray transformer that utilizes self-attention mechanisms to compose coordinate-wise point features along traced rays. This departure allows for inducing view synthesis in a more effective manner. 2) To balance memory usage and accuracy, we leverage scene-adaptive neural point clouds for scene representation [42] . Each neural point encapsulates both geometric and color features of the local map. During rendering, we employ scene-adaptive ray-based rendering facilitated by learned attention mechanisms, enabling the translation of these features into scene geometry and color estimates. 3) By utilizing the depth and color images rendered from the ray transformer outputs, we optimize scene representations and camera poses by minimizing re-rendering losses throughout the tracking and mapping processes. Our method is extensively evaluated across synthetic and real-world datasets, showcasing advancements in tracking, mapping, and rendering. In summary, the contributions of this paper are three-fold:(1) We propose SAR-SLAM, a novel neural implicit SLAM system designed to operate in real-time and demonstrate robustness across diverse challenging scenarios of various scales. (2) SAR-SLAM's core innovation lies in the implicit attentionbased ray rendering idea. Differing from the traditional volume rendering equation, this approach reconstructs NeRF without requiring an explicit rendering formula, thereby enhancing strength, scalability, and versatility in graphical rendering. (3) Comprehensive evaluations performed across various datasets validate the superiority of our approach in terms of tracking, mapping, and rendering capabilities. 
related work: 
 Dense Visual SLAM. In recent years, visual SLAM has seen significant activity in both academia and industry. Unlike sparse visual SLAM algorithms that utilize sparse point clouds for scene map representation and camera pose estimation, dense visual SLAM algorithms leverage dense point clouds, meshes, or accelerated grids to reconstruct detailed scene maps. Generally, map representations fall into two categories: view-centered and world-centered. The former often represents 3D geometry as depth maps of keyframes, as seen in approaches like DTAM [22] , ORB-SLAM [21] , and subsequent works. DTAM's straightforward pipeline is widely used in SLAM systems employing deep learning for depth and pose estimation. Similarly, DeepFactors [9] simplify optimization by employing a collection of basic depth maps. Other methods, such as CodeSLAM [2] , SceneCode [49] , and NodeSLAM [35] , optimize latent representations decoded into keyframes or object depth maps. DROID-SLAM [36] utilizes regression optical flow for geometric refinement, while TANDEM combines multi-view stereo with DSO for real-time dense SLAM. DeepSLAM [17] utilizes autoencoder networks and recurrent convolutional neural networks to predict scene depth and 6DoF pose, respectively. Alternatively, another approach employs a world-centered map representation that anchors 3D geometry in unified world coordinates, representing scene maps as surfels [30, 41] or occupancies/TSDF values in voxel grids [11, 24] . NeRF-based Visual SLAM. Thanks to neural implicit representations facilitating object-level reconstruction [46] , scene completion [19, 25] , new view synthesis [26] , and various other aspects, several NeRF-based Visual SLAM methods aim to jointly optimize the neural radiance field and camera pose. BARF [18] utilizes a neural rendering network for implementing Bundle Adjustment (BA), a crucial process in traditional SLAM systems, executing iterative optimizations for both model and camera pose. iMAP [34] utilizes the neural rendering model to establish two threads: tracking and mapping. The tracking thread leverages current model parameters to reason and optimize the camera pose of the current frame, while the mapping thread refines the network model parameters and camera pose after integrating new keyframes. Building upon iMAP, NICE-SLAM [51] employs feature grids to encode the scene into multi-dimensional vectors and utilizes an MLP to decode the implicit representation into occupancy and color. Further advancements, such as NICER-SLAM [50] , have proposed a dense RGB-SLAM system that concurrently optimizes camera poses and multi-level neural implicit representations, enabling high-precision positioning and the synthesis of new views with exceptional fidelity. Similarly, in literature [29] , the exclusive use of RGB images as input introduces a photometric consistency transformation error based on multi-view geometric constraints, resulting in enhanced constraints for camera pose estimation and the geometric structure of the scene. Unlike the NeRF-based Visual SLAM approaches mentioned earlier that rely on classic volume rendering for view synthesis subsequent to feature encoding or aggregation, our proposal introduces a self-attentive ray transformer to model this process. This methodology constructs a more versatile imaging model for the SLAM system, capable of synthesizing higher-quality images within restricted viewing angles. Consequently, it enhances the accuracy of localization and mapping of NeRF-based SLAM algorithms. Diverging from methods like NICE-SLAM that utilize multi-scale feature grids to represent the scene, our approach involves neural point cloud encoding [28, 42] . The neural points offer a user-friendly representation, facilitating faster neighborhood search and achieving a balanced trade-off between accuracy and efficiency. Upon receiving each new RGB-D image, we incorporate a set of neural points using a specific sampling approach. Subsequently, we employ a versatile ray transformer to render depth and color maps. The sampled point features encompass three components: color and geometry features derived from the neural point cloud map interpolation, and positional features from learnable Gaussian position encoding. Following the generation of rendered images (depth and RGB), we estimate camera poses and refine the scene representation through an iterative optimization process that minimizes re-rendering losses. This iterative process involves the optimization of neural point features and the ray transformer network during mapping. In an alternating manner, we optimize camera poses during tracking while keeping the map and network fixed. 
method: 
 The overview of our proposed method is depicted in Fig. 2 . We use neural point clouds to represent the scene, which is incrementally added during the exploration process (Sec. 3.1). In contrast to previous NeRF-based SLAM approaches that optimize scene representation using fixed volume rendering equations, SAR-SLAM leverages learnable self-attentive ray transformers for view synthesis (Sec. 3.2). Throughout mapping and tracking, we minimize re-rendering losses related to depth and color. This simultaneous optimization enhances both camera poses and scene representation (Sec. 3.3 ).Differing from the majority of NeRF-based SLAM methods that use a hierarchical grid for scene representation, we integrate geometry and color features into the neural point cloud using scene-adaptive point density, as proposed in [42] . We define a set of neural point clouds with 𝑁 points as follows:EQUATIONwhere 𝑝 𝑖 ∈ R 3 is the location of the anchored point, 𝑓 𝑔 𝑖 ∈ R 32 and 𝑓 𝑐 𝑖 ∈ R 32 are the geometric and color feature descriptors respectively. During the mapping process, we conduct uniform sampling and large gradient pixel sampling on RGB images. If the depth value corresponding to the sampled pixel is valid, the 2D pixel is projected onto the 3D space. Subsequently, neighboring neural points within the specified search radius are identified within the neural point cloud. The density of the neural point cloud is intricately linked to the chosen search radius, which is governed by the color gradient-based dynamic resolution strategy outlined in [28] .In scenarios where no neural point is found in the vicinity, we sample three points aligned with the depth value 𝐷 along the camera ray: (1 -𝜌)𝐷, 𝐷, and (1 + 𝜌)𝐷, where 𝜌 ∈ (0, 1) serves as a hyperparameter, accounting for the anticipated depth noise. The feature vectors of these newly added neural points are initialized using Kaiming initialization [12] . As more frames are processed, the neural point cloud expands progressively to encompass the exploration of the scene. It tends to converge to a finite set of points once all parts of the scene have been accessed. Unlike many gridbased scene representations, this approach doesn't necessitate the specification of scene boundaries before reconstruction.Current NeRF-based SLAM methods employ a fixed volumetric rendering pipeline to synthesize novel view images for poses and scene representation optimization. This pipeline combines the color and density of points along each ray cast from the image plane to produce the final pixel color. Given camera intrinsic parameters and the estimated camera pose, we can obtain the camera ray r = (o, d), where o is the origin and d is the ray direction. We sample a set of points 𝑥 𝑖 along this ray as:EQUATIONwhere 𝑧 𝑖 ∈ R is the point depth. For pixels with valid depth value 𝐷, we sample 5 points uniformly between the limited band (1 -𝜌)𝐷 and (1 + 𝜌)𝐷. With this scene depth prior, we can sample fewer samples along the ray, which achieves a computational speed-up during rendering.The volume rendering can be conceptualized as a weighted aggregation of all point-wise outputs, where the weights are globally determined by points along traced rays for occlusion modeling. We argue that this aggregation process can be effectively learned by a transformer model, referred to as the ray transformer in this paper. Specifically, the point-wise colors are mapped to token features, while the attention scores correspond to transmittance, representing the blending weights in the rendering process.The detailed network architecture of the ray transformer is depicted in Fig. 3 . Following the point sampling, we combine the EQUATIONwhere MP represents the mean pooling operation. For each point 𝑥 𝑖 , we find its closest eight neighbor points {𝑝 𝑘 , 𝑘 = 1, . . . , 8} using the corresponding per-pixel query radius, and use inverse squared distance weighting for features trilinear interpolation:EQUATION)EQUATIONwhere 𝑓 𝑔 𝑘 and 𝑓 𝑐 𝑘 are geometry features and color features of neural point 𝑝 𝑘 respectively, F 𝜃 is a one-layer MLP inspired by [42] . For pixels without valid depth, we sample 30 points along the ray within a depth interval. As an additional output of the ray transformer, the weights 𝛼 𝑖 of each predicted token can be used for depth D rendering and variance Ŝ𝐷 computation along the ray as:EQUATIONwhere 𝑁 𝑠 = 5 for sampled pixels with valid depth and 𝑁 𝑠 = 30 for invalid situation. This framework demonstrates superiority in view synthesis compared to other approaches lacking a self-attentive scheme (e.g., pure MLP. See Table 9 in Sec. 4.4).In this section, we delve into the optimization specifics concerning the color features and geometry features linked to the neural point cloud. Additionally, we cover position features encoded through learnable Gaussian positional encoding, along with insights into the ray transformer network. Implicit Mapping. In the mapping process, we conduct uniform sampling of 𝑀 pixels from the present RGB-D frame and the chosen keyframes. The re-rendering loss 𝐿 𝑚𝑎𝑝 , comprises geometric loss 𝐿 𝑔 and photometric loss 𝐿 𝑝 . Both losses are formulated as L 1 loss function:EQUATIONwhere D𝑚 and Î𝑚 are the rendered depth and the color value for given ground truth 𝐷 𝑚 and 𝐼 𝑚 , 𝜆 𝑚𝑎𝑝 is the loss weighting factor for mapping. This loss function optimizes the geometric and color features, as well as the parameters of the ray transformer and interpolation decoder simultaneously. To ensure effective initialization of the mapping optimization, the iteration count for the first frame is set considerably higher than subsequent frames. Instead of directly assigning iteration parameters, we utilize the difference in Peak Signal-to-Noise Ratio (PSNR) between the rendered image and the ground truth as a criterion. This adaptive thresholding allows us to dynamically adjust iteration numbers for different scenes, resembling the initialization process in classical SLAM systems. Camera Tracking. In a parallel process to mapping, we execute frame tracking to optimize the camera pose {R, t} for the current frame. We sample 𝑀 𝑡 pixels from the current frame and employ a modified re-rendering loss, distinct from Eq. ( 7), as follows:EQUATIONwhere Ŝ𝐷 is the standard deviation of the depth prediction, 𝜆 𝑡𝑟𝑎𝑐𝑘 denotes the loss weighting factor for tracking. For simplicity, we initialize the new pose based on a constant speed assumption widely adopted in existing methods [34, 51] . This assumption involves transforming the last known pose using the relative transformation between the second-to-last pose and the last pose. Keyframe Selection. In line with many NeRF-based SLAM systems, we utilize a keyframe database to regulate the mapping loss. We select a subset of keyframes that demonstrate significant overlap with the viewing frustum of the current frame, integrating pixel samples from these keyframes. This selection method ensures efficient optimization for the neural points within the current view and maintains geometric consistency throughout mapping. To begin, we project all neural points onto the current frame using the optimized camera pose to facilitate effective neural point selection. Additionally, we project the neural points onto every keyframe in the global keyframe list and arrange all keyframes in descending order based on the number of projected neural points. Subsequently, we select the top 𝐾 -1 keyframes from the sorted list to complement the current frame, resulting in a total of 𝐾 active frames for local bundle adjustment.The detailed tracking and mapping process of SAR-SLAM is illustrated in Alg. 1. Following system initialization, the tracking and mapping operate simultaneously, optimizing camera poses and scene representation alternatively. Features extraction; ⇒ Eq. ( 4), Eq. ( 5)RGB and depth image rendering; ⇒ Eq. ( 3), Eq. ( 6)8Construct tracking loss 𝐿 𝑡𝑟𝑎𝑐𝑘 ; ⇒ Eq. ( 8)9Optimize camera pose T 𝑗 ; Construct mapping loss 𝐿 𝑚𝑎𝑝 ; ⇒ Eq. ( 7)13Optimize 𝑃 and ray transformer parameters;  
experimental setup: 
 Datasets. The synthetic dataset Replica comprises high-quality 3D reconstruction of a variety of indoor scenes. We utilize the publicly available dataset collected by Sucar et al [34] ., which provides ground truth trajectories and RGBD sequences. Further, we evaluate the performance of our framework in real-world scenes by using TUM-RGBD [33] and the ScanNet [10] dataset. The poses for TUM-RGBD were captured using an external motion capture system while ScanNet uses poses from BundleFusion [11] . Baseline Methods. We primarily compare our proposed method against existing state-of-the-art NeRF-based visual SLAM methods, including NICE-SLAM [51] , Vox-Fusion [44] , ESLAM [14] , Uncle-SLAM [29] , and Point-SLAM [28] . We derive most comparison results directly from respective papers. For datasets or sequences not available, we reproduce the results using open-source code with default settings. Evaluation Metrics. For tracking, we follow the conventional SLAM evaluation pipeline by aligning the estimated camera trajectory to the GT and using ATE RMSE to evaluate the accuracy. To evaluate scene reconstruction, we produce the meshes by marching cubes and adopt the F-score (harmonic mean of the Precision and Recall) and depth L1 as the primary metrics with a distance threshold of 1𝑐𝑚. Furthermore, we provide PSNR, SSIM, and LPIPS for rendering evaluation, calculated based on rendering the fullresolution image along the estimated trajectory every 5𝑡ℎ frame. Implementation Details. We executed our SLAM system on a desktop PC equipped with a 3.80GHz Intel i9-12900K CPU and an NVIDIA RTX 3090 Ti GPU. For small-scale synthetic datasets (Replica), we select 𝐾 = 5 keyframes for local bundle adjustment, 𝑀 = 5, 000 pixels for mapping, and 𝑀 𝑡 = 1, 500 pixels for tracking.Conversely, for large-scale real datasets (ScanNet and TUM-RGBD), we adjusted the parameters to 𝐾 = 10, 𝑀 = 10, 000 and 𝑀 𝑡 = 5, 000. Rather than uniformly sampling pixels, we employed a sceneadaptive scheme that selectively samples half of the pixels based on the image gradient magnitude. Throughout all experiments, we maintained a photometric loss weighting of 𝜆 map = 0. Tracking Evaluation. The tracking results on the Replica dataset are reported in Table 1 . SAR-SLAM emerges as the superior performer among all comparison methods across the test sequences. We attribute this success to the introduction of the self-attentive ray rendering and the notably enhanced scene representation afforded by the neural point cloud. Moving to Table 2 , the performance of our proposed method on the real-world TUM-RGBD dataset is presented. Outperforming all existing NeRF-based methods, our approach showcases remarkable results. For the more intricate real-world ScanNet dataset, our method achieves superior tracking results on average compared to existing NeRF-based methods as shown in Table 3 . However, it's important to note that the neural point cloud exhibits sensitivity to motion blur and specularity, leading to a noticeable degradation in Point-SLAM's performance in selected sequences. In these complex scenes, our method surpasses Point-SLAM due to the incorporation of the learnable ray transformer, which mitigates these challenges more effectively. Mapping Evaluation. Table 4 presents a comparison between our method and several others, including NICE-SLAM [51] , VoxFusion [44] , Point-SLAM [28] , and ESLAM [14] , focusing on reconstruction accuracy. Across all evaluation metrics, our method outperforms all others. Specifically, we showcase an average enhancement of 87%, 84%, 11%, and 67% on the depth L1 metric in comparison to NICE-SLAM, Vox-Fusion, Point-SLAM, and ESLAM respectively. In Fig. 4 , we provide a visual comparison of the mesh reconstructions generated by NICE-SLAM and our method against the ground truth. Notably, our method exhibits a significant improvement by reconstructing scene maps with higher accuracy and clarity, presenting finer details compared to previous approaches. This enhancement in reconstruction quality owes itself to two key components: the ray transformer, capable of dynamically adjusting the attention distribution to finely control the sharpness of the reconstructed surface, and the neural point cloud, which adeptly adjusts point density.Rendering Evaluation. Table 5 presents a comparison of rendering results, demonstrating the superiority of our method over existing NeRF-based SLAM approaches. Additionally, in Fig. 5 , Fig. 6 (a), and Fig. 6 (b), we showcase exemplary novel view synthesis, highlighting how our proposed SAR-SLAM generates more accurate details.We present a breakdown of the runtime and memory utilization within Table 6 for the Replica Office 0 scene. The tracking and mapping runtimes are detailed per iteration and frame. Our method exhibits a significantly smaller GPU memory footprint compared to NICE-SLAM, and slightly smaller than Point-SLAM. These runtimes were evaluated using a single Nvidia RTX 3090 Ti GPU. Furthermore, the running time and GPU/RAM memory footprint in various scenarios with different configurations are presented in the following Table 7 . Despite the increase in sampled pixels, the corresponding increase in time and memory utilization remains acceptable. System Ablation. We assess the performance of our proposed SAR-SLAM under various system configurations. Table 8 presents performance metrics obtained from the Replica Office 0 dataset. Utilizing ray transformer significantly enhances tracking, mapping, and rendering performance, underscoring the efficacy of selfattentive rendering for SLAM systems. Furthermore, employing the keyframe selection (KS) scheme and local bundle adjustment (BA) for mapping optimization leads to a further reduction in localization error. Moreover, comparing Sample Points per Ray (SPR) reveals that 5 points suffice to ensure accuracy. Network Ablation. In Table 9 , we present a comparison of our method across various network settings, as depicted in Fig. 3 . Our proposed method (Full) outperforms the variant without the selfattention module (w/o SA), demonstrating the effectiveness of the self-attentive rendering. When changing the multi-layer architecture in the ray transformer into a single transformer layer (w/o ML), we observe a decrease in evaluation metrics to varying degrees. Additionally, excluding the Feed-Forward Network (w/o FFN) or Feature Concatenation (w/o FC) leads to less accurate rendering of depth and RGB images, consequently impacting the precision of tracking and mapping Keyframe Selection Ablation. We evaluated our method using the keyframe selection strategies from iMAP and Point-SLAM, as shown in Table 10 . iMAP's KS strategy randomly selects keyframes from the global scene to mitigate the scene-forgetting problem. In contrast, Point-SLAM's KS strategy maintains the keyframe list by evaluating view overlap, resulting in improved tracking accuracy and faster convergence. Unlike Point-SLAM, which utilizes sample pixels for overlap computation, our method employs the projection of neural points to obtain more effective pixels for bundle adjustment optimization, thereby enhancing tracking performance. Adaptive Initialization Ablation. Fig. 7 illustrates a comparison of geometry optimization across various iteration settings for mapping initialization. Our proposed adaptive initialization method demonstrates superior performance in achieving a balance between running time and scene representation accuracy when compared to a fixed iteration number.  
conclusion: 
 We introduce SAR-SLAM, a novel neural implicit SLAM system that integrates a ray transformer for view rendering and neural point cloud encoding for scene representation. The ray transformer synthesizes novel views via a self-attentive mechanism that decodes a sequence of point features sampled along the camera ray into pixel color and blending weight, enabling end-to-end learning and eliminating the necessity for fixed volume rendering equations. Experimental results on both synthetic and real-world scenes consistently demonstrate the superior performance of SAR-SLAM over existing NeRF-based SLAM methods. Notably, it exhibits enhanced tracking, reconstruction, and rendering accuracy, while also showcasing superior runtime and memory efficiency. 
 
 Appendix:  


Decision: The paper introduces a new approach to simultaneous localization and mapping which addresses some limitations (reliance on volume rendering techniques, which oversimplify the modelling) of existing NeRF-based SLAM approaches, leading to improved accuracy and efficiency.

The rebuttal addressed the main concerns raised by reviewers.

Reviews: review: paper_summary: paper_summary: This paper presents a simple but effective idea in the field of neural implicit SLAM: combining Point-SLAM with an IBR-style architecture that refines point features with cross attention. With the major contribution centered around the mapping part, the tracking part largely follows existing pipelines. Evaluations on common public benchmarks like Replica are presented showing improved tracking and mapping performance. Surprisingly, although this architecture adds new componenets to Point-SLAM, the efficiency is improved in terms both speed and storage. This involves good engineering efforts and I am looking forward to the code.
strength_weakness: strength_weakness: + The idea is simple but very effective as shown by the experiments. Interpolation in point based representation may be suboptimal and using a data-driven attention-based module to mix features from the neighborhood makes sense. This leads to a combination of Point-SLAM and IBR but this design differs from IBR in nature. IBR mixes features from views while this one mixes features from the neighbors. This is considered as a novel usage of an existing module in another (reasonable) setting.
+ The experiments are quite comprehensive, showing good ablations and benchmark results. A notable thing is that while this design incorpoarates new modules into Point-SLAM, efficiency in terms of both speed and storage are improved. The field of SLAM naturally favors this kind of engineering efforts.
limitations: limitations: - The effectiveness of this method seems to originate from the principle idea: using IBR attention to improve the feature mixing in a Point-NeRF. I think this is not specific to the SLAM setting and also applies to the original Point-NeRF setting. Could you please provide another ablation study that investigates whether this architecture works for generic novel view synthesis quality of Point-NeRF or Point-NeRF variants? I think this is important to reach the conclusion how this paper brings knowlege to the whole neural rendering community.
- A notable advantage of this paper its efficiency caused by good engineering. But for now, the (anonymous) code is not released yet, which is a pity. This is an important factor for SLAM papers.
- I recommend two references on neural rendering effiency [A] and multis-cale rendering quality [B], which is related to the paper. Could you please kindly consider include and discuss them in the paper?

[A] SlimmeRF: Slimmable Radiance Fields, 3DV 2024

[B] Rip-NeRF: Anti-aliasing Radiance Fields with Ripmap-Encoded Platonic Solids, SIGGRAPH 2024
paper_summary: paper_summary: This paper propose SAR-SLAM, a point cloud encoding-based and rendering-based SLAM. Compared with previous methods, the proposed methods utilize reconstructed NeRFs and represent the scene through point cloud encoding. The proposed method achieved better performance than previsou methods on various dataset.
strength_weakness: strength_weakness: 1) The writing is easy to follow and understand.
2) The experiments are sufficient.
limitations: limitations: 1) The novelty is limited. The proposed method is similar to Point-NeRF, which is not cited in the paper. The proposed method follows previous methods, Nice-slam and point-slam, and just replace the rendering method in the pipeline with Point-NeRF. So I think the contribution is not sufficient.
2) I wonder how to get the initial point clouds? The initila point clouds are important in the rendering and localization. Are the initial point clouds robust to the noise and outliers?
3) In the implicit mapping, should the query image be close to the target image?
4) The self attention is time and memory consuming. Is there efficiency analysis?
paper_summary: paper_summary: SAR-SLAM introduces a novel approach to simultaneous localization and mapping by integrating a ray transformer for view rendering and neural point cloud encoding for scene representation. Departing from traditional methods, it addresses limitations of existing NeRF-based SLAM approaches, offering enhanced accuracy and efficiency. Through self-attentive rendering, SAR-SLAM enables end-to-end learning, eliminating the need for fixed rendering equations. Experimental results underscore its superiority, marking a significant advancement in SLAM research.
strength_weakness: strength_weakness: 1. The paper exhibits clear and well-organized presentation, ensuring ease of comprehension.
2. Illustrations and figures effectively elucidate the discussed concepts, aiding in the understanding of the paper's content.
3. SAR-SLAM's adoption of an implicit attention-based ray rendering approach represents a notable innovation, enhancing the system's strength, scalability, and versatility in graphical rendering.
4. Qualitative results convincingly demonstrate the effectiveness of the proposed method, highlighting its potential in SLAM applications.
limitations: limitations: 1. The absence of outdoor and more challenging scene data limits the comprehensive evaluation of SAR-SLAM's performance across diverse environments.
2. A comparison with NeRF-based SLAM and 3DGS-based SLAM methodologies would provide valuable insights into SAR-SLAM's comparative advantages.
> Matsuki H, Murai R, Kelly P H J, et al. "Gaussian splatting SLAM." arXiv preprint arXiv:2312.06741, 2023.