Title: CP-SLAM: Collaborative Neural Point-based SLAM System
Venue: NeurIPS 2023
Accepted: yes
Keywords: Collaborative SLAM; Neural Point Field; Keyframe-based SLAM; Pose Graph Optimization
Abstract: This paper presents a collaborative implicit neural simultaneous localization and mapping (SLAM) system with RGB-D image sequences, which consists of complete front-end and back-end modules including odometry, loop detection, sub-map fusion, and global refinement. In order to enable all these modules in a unified framework, we propose a novel neural point based 3D scene representation in which each point maintains a learnable neural feature for scene encoding and is associated with a certain keyframe. Moreover, a distributed-to-centralized learning strategy is proposed for the collaborative implicit SLAM to improve consistency and cooperation. A novel global optimization framework is also proposed to improve the system accuracy like traditional bundle adjustment. Experiments on various datasets demonstrate the superiority of the proposed method in both camera tracking and mapping.

CP-SLAM: Collaborative Neural Point-based SLAM

Abstract

This paper presents a collaborative implicit neural simultaneous localization and mapping (SLAM) system with RGB-D image sequences, which consists of complete front-end and back-end modules including odometry, loop detection, sub-map fusion, and global refinement. In order to enable all these modules in a unified framework, we propose a novel neural point based 3D scene representation in which each point maintains a learnable neural feature for scene encoding and is associated with a certain keyframe. Moreover, a distributed-to-centralized learning strategy is proposed for the collaborative implicit SLAM to improve consistency and cooperation. A novel global optimization framework is also proposed to improve the system accuracy like traditional bundle adjustment. Experiments on various datasets demonstrate the superiority of the proposed method in both camera tracking and mapping.

1 Introduction

Dense visual Simultaneous Localization and Mapping (SLAM) is an enduring fundamental challenge in computer vision, which aims to achieve 3D perception and exploration of unknown environments by self-localization and scene mapping with wide downstream applications in autonomous driving, unmanned aerial vehicle(UAV) navigation, and virtual/augmented reality (VR/AR).

The traditional visual SLAM has witnessed continuous development, leading to accurate tracking and mapping in various scenes. Most of the current prominent visual SLAM systems pay their primary attention to real-time tracking performance [29, 31, 4], while dense map reconstruction is normally achieved with the temporal fusion of additional depth input or estimated depth images, thus sensitive to noises and outliers. Some collaborative visual SLAM systems [34, 20] have also been proposed as a straightforward extension of monocular visual SLAM.

The learning-based visual SLAM has attracted more attention recently with better robustness against noises and outliers [37]. Very recently, some methods [36, 50, 44] exploit the Neural Radiance Fields (NeRF) for dense visual SLAM in a rendering-based optimization framework showing appealing rendering quality in novel view. However, different from the traditional feature-based SLAM system, such learning-based methods with implicit representation are normally pure visual odometry systems without loop closure and pose graph optimization due to the limitation of the scene representation (e.g., a neural network or feature grid), which also makes it hard to be adapted to the collaborative SLAM. Take the feature grid representation as an example, it is hard to adjust or align the feature grid when the pose is optimized after loop closure or transformed to the unified global coordinate system for collaborative SLAM.

In this paper, we introduce a novel collaborative neural point-based SLAM system, named CP-SLAM, which enables cooperative localization and mapping for multiple agents and inherently supports loop closure for a single agent. However, it is nontrivial to design such a system. At first, we need a new neural representation for SLAM that is easy to be adjusted for loop closure and collaborative SLAM. Inspired by Point-NeRF [43], we built a novel neural point-based scene representation with keyframes.

The scene geometry and appearance are encoded in 3D points with per-point neural features, and each point is associated with a certain keyframe. In this way, when the camera poses are optimized with loop closure and pose graph optimization, these neural points can be easily adjusted like traditional 3D points. Second, different from the monocular implicit SLAM system, a new learning strategy is required for collaborative implicit SLAM. For better consistency and cooperation, we present a two-stage learning strategy, i.e., distributed-to-centralized learning. In the first stage (before sub-map fusion), we set up an independent group of decoders for each RGB-D sequence and update them separately. In the second stage (after sub-map fusion), we fuse weights from all groups of decoders and continue with lightweight fine-tuning, after which all sequences can share and jointly optimize a common group of decoders. Furthermore, like the bundle adjustment in traditional SLAM, a novel optimization framework is needed to adjust both the camera poses and scene geometry for the neural implicit SLAM system. To this end, we introduce the pose graph optimization into the implicit SLAM system followed by a global map refinement.

Our contributions can be summarized as follows. At first, we present the first collaborative neural implicit SLAM system, i.e., CP-SLAM, which is composed of neural point based odometry, loop detection, sub-map fusion, and global refinement. Second, we propose a new neural point 3D scene representation with keyframes, which facilitates map fusion and adjustment. Furthermore, novel learning and optimization frameworks are proposed to ensure consistent and accurate 3D mapping for cooperative localization and mapping. We evaluate CP-SLAM on a variety of indoor RGB-D sequences and demonstrate state-of-the-art performance in both mapping and camera tracking.

2 Related Work

Single-agent Visual SLAM . With the development of unmanned intelligence, visual SLAM becomes an active field in the last decades. Klein et al. proposed a visual SLAM framework [15] that separates tracking and mapping into different threads. This framework is followed by most of current methods. Traditional single-agent visual SLAM uses filtering or nonlinear optimization to estimate the camera pose. Filtering based methods [27, 3, 23] are more real-time capable but less globally consistent. In contrast, optimization-based [28, 29, 31, 4, 8, 19] methods can make full use of past information. Meanwhile, multi-source information is widely incorporated into the SLAM system to improve specific modules, such as light-weight depth estimation [48, 42].

Collaborative Visual SLAM . Collaborative SLAM can be divided into two categories: centralized and distributed. CVI-SLAM [14], a centralized visual-inertial framework, can share all information in a central server and each agent outsources computationally expensive tasks. In centralized SLAM, the server manages all sub-maps, performs map fusion and global bundle adjustment, and feeds processed information back to each agent. This pipeline is reproduced in CCM-SLAM [33], where each agent is equipped with a simple visual odometry and sends localization and 3D point cloud to the central server. In terms of distributed systems, Lajoie et al. developed a fully distributed SLAM [18], which is based on peer-to-peer communication and can reject outliers for robustness. NetVLAD [1] is used in [18] to detect loop closures. In addition, [10] proposed the compact binary descriptor specifically for multi-agent system. Compared with traditional collaborative systems, our system can perform dense mapping with fewer neural points.

Neural Implicit Representation . Neural implicit field showed outstanding results in many computer vision tasks, such as novel view synthesis [24, 25, 38, 2], scene completion [5, 11, 30] and object modelling [6, 22, 40, 45, 46]. In recent study, some works [47, 41] attempted to reversely infer camera extrinsics from the built neural implicit field. Inspired by different representations of neural field including voxel grid [21] and point cloud [43], NICE-SLAM [50] and Vox-Fusion [44] chose voxel grid to perform tracking and mapping instead of a single neural network which is limited by expression ability and forgetting problem. Both of them are most related works to ours. Besides, vMAP [16] can efficiently model watertight object models in the absence of 3D priors. ESLAM [13] turns to tri-plane and Truncated Signed Distance Field (TSDF) to solve RGB-D SLAM. In addition to purely neural-based methods, some hybrid systems, such as NeRF-SLAM [32] and Orbeez-SLAM [7], construct the neural map but use traditional methods to estimate poses.

3 Method

The overview of our collaborative SLAM system is shown in Fig. 1. Given a set of RGB-D sequences, our system incrementally performs tracking and mapping based on neural point cloud representation for each agent (Section 3.1). We incorporate a learning-based loop detection module that extracts unique descriptors for 2D frames, and stitches sub-maps through high-quality loop constraints

Sequence-1

𝑝𝑖

, 𝑓

𝑖

𝑪

𝒇𝒊,𝒙 𝒊

𝑼

𝒄

𝑮

𝝈

Σ

Sequence-2

NetVLAD

Descriptor Pool

𝝀𝒇𝒊𝒏𝒆

Sub-map Fusion

Keyframe-centric

Map Refinement

PGO

Depth Loss

Color Loss

𝑥𝑖

Q00-0

SimiliarProcessasSequence-1

Similiar

ProcessasSequence-1

(Section 3.2). We further design a two-stage (distributed-to-centralized) MLP training strategy to improve consistency and strengthen collaboration (Section 3.3). To reduce cumulative error in mapping and tracking, we use co-visibility among sub-maps to take global pose graph optimization as the back-end processing, followed by frame-based map refinement (Section 3.4). We will elaborate on the entire pipeline of our system in the following subsections.

3.1 Neural Point based Odometry

Our system starts from a front-end visual odometry module, in which we combine pose backpropagation update and point-based neural field to perform sequential tracking and mapping. Point-based Neural Field. We divide an image into 4 × 4 patches and incrementally project each central pixel of every patch to its corresponding 3D location p ∈ R 3 . To acquire the feature embedding f ∈ R 32 anchored on p , a 2D image is fed into an untrained single-layer convolutional neural network. We define our neural point cloud as:

Volume Rendering. We emulate the differentiable volume rendering strategy in NeRF [26], where radiance and occupancy are integrated along rays to render color and depth maps. Given a 6DoF pose { R c , O c } and intrinsics of a frame, we can cast rays from randomly selected pixels and sample N total points. Any sampling point is defined as

where t i ∈ R is the depth of the point and d ∈ R 3 is the unit ray direction. Guided by depth information, we can distribute our samples near the real surface. Specifically, we sample N total = N near + N uni points on each ray. For any pixel with valid depth D , t i is uniformly sampled within the intervals [ 0 . 95 D , 1 . 05 D ] and [ 0 . 95 D min , 1 . 05 D max ] respectively, yielding N near and N uni sample points correspondingly, where D min and D max are minimum and maximum depth values of the current depth map. For any pixel without valid depth values, t i is uniformly sampled within the interval [ D l , 1 . 05 D max ] , generating N total points. For each point x i , we firstly query K neighbors { p k | k = 1 , ..., K } within query radius r and then use an MLP C to convert the original neighbor feature f k into f k,x i that incorporates relative distance information, i.e.,

Then a radiance MLP U , decodes the RGB radiance c at location x i using an interpolation feature f x i , which is obtained from weighted interpolation according to inverse distance weights and neighboring features:

If no neighbors are found, the occupancy σ at x i is set to zero. Otherwise, we regress σ using an occupancy MLP G at x i . We follow a similar inverse distance-based weighting interpolation as radiance, instead of interpolating at feature level. We decode σ i for each neighbor and finally interpolate them:

Next we use c x i , σ x i regressed from MLPs to estimate per-point weight α x i . α x i is regarded as the opacity at x i , or the probability of a ray terminates at this point and z x i is the depth of point x i . Depth map and color map can be rendered by calculating depth and radiance expectations along rays as in Eq. 8.

Mapping and Tracking. We use rendering difference loss as described in Eq. 9 that consists of geometric loss and photometric loss during the mapping process. For a new-coming frame, we sample M 1 pixels to optimize point-anchored features f i and parameters of MLP C , U , G . With the first frame, we need to perform a good initialization at a few higher cost of M 3 pixels and around 3000 ∼ 5000 optimization steps to ensure smooth following processing. For subsequent mapping, we select pixels uniformly from the current frame and 5 co-visible keyframes. We find that joint mapping can effectively stabilize the tracking process.

where D m , I m represent ground truth depth and color map, ˆ D m , ˆ I m are corresponding rendering results, and λ 1 is the loss balance weight. During the tracking process, we backpropagate farther to optimize camera extrinsics { q, t } while keeping features and MLPs fixed, where q is quaternion rotation and t is the translation vector. We sample M 2 pixels across a new-coming frame and assume the zero motion model where the initial pose of a new frame is identical to that of the last frame. Considering the strong non-convexity of the color map, only the geometry part is included in tracking loss:

3.2 Loop Detection and Sub-Map Alignment

To align sub-maps and reduce accumulated pose drift, we perform loop detection and calculate relative poses between different sub-maps. For each keyframe in the sub-map, we associate it with a descriptor generated by the pre-trained NetVLAD [1] model. We use cosine similarity between descriptors as the judgment criteria for loop detection. When the camera moves too much between frames, pose optimization tends to fall into local optimum, so we use two similarity thresholds λ fine , λ coarse to find matching pairs that have enough overlap and ensure correct loop relative pose. Given two sub-maps, M 1 and M 2 , we find out loop frames with largest overlap, { I l 1 , I l 2 } , with similarity greater than λ fine , and small overlap loop frames, { ( I s 1 1 , I s 2 2 ) , ..., ( I s 1 n , I s 2 n ) } , with similarity greater than λ coarse but less than λ fine . For pairs { I l 1 , I l 2 } , we take pose of I l 1 as initial pose and I l 2 as the reference frame. The tracking method described in 3.1 can be used to obtain the relative pose measurement between {M 1 , M 2 } . We can use this relative pose to perform a 3D

Fusion

Feature grid misaligned

x

i

Free fusion

Agent 1

Agent 2

(a)

(b)

Fusion

rigid transformation on sub-maps, so that all sub-maps are set in the same global coordinate system, achieving the purpose of sub-map alignment as shown in Fig. 2.

where T r is the relative pose between M 1 and M 2 , and D, ˆ D is the rendered depth and ground truth depth. For loop frames with small overlap { I s 1 i , I s 2 i } , we only use them to perform pose graph optimization (refer to 3.4). Sub-map fusion will lead to neural point redundancy in specific areas, which imposes a burden on computing and memory. Due to the sparsity of the neural point cloud, we adopt a grid-based filtering strategy, that is, we perform non-maximum suppression based on the distance from a neural point to the center of a ρ 3 cube.

3.3 Distributed-to-Centralized Learning

To enhance consistency and cooperation, in collaborative SLAM, we adopt a two-stage MLP training strategy. At the first stage (Distributed Stage), each image sequence is considered as a discrete individual with a unique group of MLPs { C j , U j , G j } for sequential tracking and mapping. After loop detection and sub-map fusion, we expect to share common MLPs across all se-

Centralized

MLP-1

Sharing MLPs

MLP-2

quences (Centralized Stage). To this end, we introduce the Federated learning mechanism which trains a single network in a cooperating shared way. At the same time as sub-map fusion, we average each group of MLPs and fine-tune the averaged MLPs on all keyframes to unify discrete domains. Subsequently, we iteratively transfer sharing MLPs to each agent for local training and average the local weights as the final optimization result of sharing MLPs, as shown in Fig. 3.

3.4 Pose Graph Optimization and Global Map Refinement

After front-end processing of all sequences, including tracking, mapping, loop detection, and sub-map fusion, we establish a global pose graph model, in which per-frame poses are nodes, sequential relative pose, and loop relative pose are edges. The pose graph model is illustrated in Fig. 4. We carry out global pose graph optimization across the entire pose graph, referring to tradi-

Keyframe-centric Model

Binding

tional visual SLAM, to force the estimated trajectory closer to the ground truth. Global pose graph

optimization effectively mitigates cumulative error and improves tracking accuracy. We use the Levenberg Marquarelt algorithm to solve this nonlinear global pose graph optimization problem described by Eq. 12, where v is the set of nodes, E s is the set of sequential edges, E l is the set of loop edges and Λ i represents the uncertainty of corresponding edges.

Naturally, it is expected that the neural point cloud layout should be rearranged following global pose graph optimization. However, a world-centric point cloud map obviously cannot allow such adjustment. To tackle this limitation, we propose a keyframe-centric neural point field, where each 3D point is associated with a keyframe (Fig. 4). Benefitting from this design, we can refine our point cloud 3D locations according to optimized poses. Following global map refinement, we employ grid-based filtering to deal with neural point cloud redundancy occurring in local regions. Considering the slight mismatch between refined neural fields and decoders after global map refinement, we end up performing a low-cost fine-tuning of the global neural point fields with fewer optimization iterations.

4 Experiments

Our CP-SLAM system supports both single-agent and multi-agent modes. Thus, we evaluate our proposed collaborative SLAM system in two aspects, both single-agent experiments with loop closure and two-agent experiments, of varying sizes and complexity. In terms of a single agent, we generate datasets based on the Replica [35] scenes and then compare our method against recent neural and traditional RGB-D SLAM methods. For the two-agent side, since no collaborative neural SLAM work has emerged so far, we compare our method with traditional methods. We also conduct ablation studies to show the importance of modules in the proposed system.

Implementation Details. CP-SLAM system runs an RGB-D sequence on an NVIDIA RTX3090 GPU. In the two-agent experiment, we need an additional RTX3090 as the central server. To encode higher frequency detail information, during inference, we impose position encoding on the origin neighbor feature f i and relative distance ∥ p i -x ∥ with an order of 1 and 7. We utilize the FRNN library to query K = 8 nearest neighbors on GPU. In all our experiments, we set N near = 16 , N uni = 4 , λ 1 = 0 . 2 , D l = 0 . 001 m,r = 0 . 15 m,ρ = 0 . 14 m,M 1 = 3000 , M 3 = 3136 , M 2 = 1500 . We extract a keyframe every 50 frames and perform map optimization and point cloud supplementation every 10 frames. For single-agent experiments, we optimize the neural field for 200 iterations. For two-agent experiments, considering that features anchored on neural points do not need to be trained from scratch after sub-map fusion, we reduce the number of iteration steps to 150. Further implementation details can be found in our supplementary material.

Baselines. In the single-agent experiment, because we use the rendered loop-closure data, we primarily choose the state-of-the-art neural SLAM systems such as NICE-SLAM [50], Vox-Fusion [44] and ORB-SLAM3 [4] for comparison on the loop-closure dataset. For the two-agent experiment, we compare our method with traditional approaches, such as CCM-SLAM [33], Swarm-SLAM [17] and ORB-SLAM3 [4].

Datasets. For reconstruction assessment, we utilize the synthetic dataset Replica [35], equipped with a high-quality RGB-D rendering SDK. We generate 8 collections of RGB-D sequences, 4 of which represent single-agent trajectories, each containing 1500 RGB-D frames. The remaining 4 collections are designed for collaborative SLAM experiments. Each collection is divided into 2 portions, each holding 2500 frames, with the exception of Office-0-C which includes 1950 frames per part.

Metrics. As a dense neural implicit SLAM system, we quantitatively and qualitatively measure its mapping and tracking capabilities. For mapping, we evaluate L1 loss between 196 uniformly-sampled depth maps, which are rendered from our neural point field, and ground truth ones. Furthermore, in terms of 3D triangle mesh, we compute mesh reconstruction accuracy. For tracking, we use ATE RMSE, Mean and Median to comprehensively measure trajectory accuracy so as to prevent the negative impact caused by a few extreme outliers.

4.1 Two-agent Collaboration

We provide the quantitative results of two-agent experiments on four scenes including Replica [35] Apartment-0, Apartment-1, Apartment-2 (multi-room), and Office-0-C (single-room). We take RGBbased CCM-SLAM [33], RGBD-based Swarm-SLAM [17] and traditional ORB-SLAM3 [4] for comparison. Table. 1 reports the localization accuracy of different methods. Despite being affected

GT

Apartment-1

Apartment-2

Apartment-0

(CCM-SLAM Failed)

Office-0-C

中

CCM-SLAM

Ours

by complex environments in multi-room sequences, the proposed system generally maintains better performance than other methods. It is worth noting that ORB-SLAM3 is not a collaborative SLAM system. It lacks the capability to process multiple sequences at the same time, thus unable to overcome the efficiency bottleneck in scene exploration. Specifically, we concatenate multiple image sequences and feed them into ORB-SLAM3, leveraging its "atlas" strategy for multi-sequence processing. Also, it can be seen that CCM-SLAM failed in some scenes because traditional RGB-based methods are prone to feature mismatching especially in textureless environments. Fig. 5 depicts the trajectories of each agent. Once two sub-graphs are fused, only a low-cost fine-tuning is required to adjust two neural fields and corresponding MLPs into a shared domain. Afterward, these two agents can reuse each other's previous observations and continue accurate tracking. Shared MLPs, neural fields, and following global pose graph optimization make CP-SLAM system a tightly collaborative system.

4.2 Single Agent with Loop

In Table. 2, we illustrate the localization performance of our system operating in single-agent mode on 4 loop closure datasets. Our method exhibits a notable superiority over recent methods, including NeRF-based and traditional ones, primarily attributed to the integration of concurrent front-end and back-end processing, as well as the incorporation of neural point representation. Qualitatively, we present trajectories of Room-0-loop and Office-3-loop from NeRF-based methods in Fig. 6. The

Office-3-loop

Room-1-loop

GT

NICE-SLAM

Ours

Vox-Fusion

YDN

experimental results demonstrate that the concentration level of density energy around sampling points is critical for neural SLAM. NICE-SLAM [50] uses dense voxel grids, which contain a large number of empty spaces, and sampling points in these empty spaces have almost no contribution to the gradient propagation. Vox-Fusion [44] has incorporated an important modification, implementing a sparse grid that is tailored to the specific scene instead of a dense grid. However, grid nodes can only be roughly placed near objects, which is unfavorable for tracking. In our point-based method, we ensure that neural field fits the real scene well. Feature embeddings can accurately include scene information. Concentratively distributed sample points and neural point based representation provide more exact gradients for pose backpropagation, which enables us to surpass NICE-SLAM and Vox-Fusion at lower resolution and memory usage. Moreover, we extend experiments on the TUM-RGBD real-world dataset, comparing with Co-SLAM [39] and ESLAM [12]. The results in Table. 3 illustrate that our method has also achieved state-of-the-art performance in the real-world setting, and the loop detection and pose graph optimization are equally effective for the real-world scene.

4.3 Map Reconstruction

The results in Table. 4 present a quantitative analysis of the geometric reconstruction produced by our proposed system in comparison to NICE-SLAM [50] and Vox-Fusion [44]. In our approach, we render depth maps and color maps every 10 frames throughout the entire predicted trajectories and utilize TSDF-Fusion (built-in function in Open3D [49] library) to construct mesh map. In

NICE-SLAM

Vox-Fusion

Ours

GT

(a)

(b)

(c)

NICE-SLAM, several metrics are employed for mapping evaluation, namely depth L1 loss, mesh accuracy, completion, and completion ratio. In our loop datasets, scenes are not completely scanned, which leads to holes in mesh reconstruction. Therefore, our comparative experiments mainly focus on depth L1 loss and mesh accuracy. Fig. 7 qualitatively compares the single-agent reconstruction results of three methods and shows our collaborative mapping results. Evidently, our method achieves more detailed geometry across all datasets.

4.4 Ablation Study

In this section, we examine some modules and designs in our system to prove their importance and the rationality of our pipeline.

Pose Graph Optimization. In this section, we conducted ablation experiments on the PGO module. Table. 1 and Table. 2 report results of ours(w / o PGO) and ours(w / PGO) in single-agent and two-agent cases respectively. With the help of PGO module, the average positioning accuracy in the single-agent experiment decreases by 10 % , while that in the two-agent experiment decreases by 13 % .

Map Refinement. As shown in Fig. 8(a), we qualitatively illustrate the neural point field layout before and after map refinement in MeshLab. We can observe that a refined neural point cloud fits the ground truth mesh better.

Sampling Concentration. The density concentration near sampling points is a keypoint that determines the performance of neural SLAM. We design a set of experiments with different sampling interval lengths on Replica Room-0-Loop sequence. As shown in Fig. 8(b), tracking accuracy and depth L1 loss consistently drop to 1.0cm and 0.6cm as sampling points gradually diverge. This experiment fully verifies the theory in Section. 4.2.

Neural Point Density. In CP-SLAM, we employed a fixed-size cubic cell within the filtering strategy

(a)

(b)

1.0

0.8

0.6

0.4

Mean Error

0.2

Depth L1 Error

0.0

0.1D

0.4D

0.6D

0.8D

to adjust the neural points, i.e., only the neural point closest to the center of a cubic cell is retained. To further explore the impact of point cloud density on tracking accuracy, we compared performance in the original Replica Room0 scene across various sizes of the cubic grid. The ablation study results in Table. 5 demonstrate that when the number of neural points is small, they are not enough to encode detailed scene geometry and color due to inadequate scene representation. Conversely, an excessive number of neural points obviously extends the learning time to converge. We empirically found that the setting ρ = 14 cm worked consistently well on all the test scenes with different complexities in our experiment.

Memory and Runtime Analysis. We evaluated the runtime and memory consumption of our system on Replica Office-0-loop scene compared to NICE-SLAM and Vox-Fusion. We report single-frame tracking and mapping time, the size of MLPs and memory footprint of the whole neural field in Table. 6. The huge feature size in NICE-SLAM is due to its dense hierarchical feature grid.

5 Conclusion

We have proposed CP-SLAM, the first dense collaborative neural SLAM framework based on a novel neural point based representation, which maintains complete front-end and back-end modules like traditional SLAM systems. The comprehensive pipeline enables our system to outperform the state-of-the-art methods in both localization and reconstruction. One limitation of our method is its requirement for considerable GPU resources to operate multiple image sequences. Also, our system has slightly weaker hole-filling ability in unobserved regions than feature grid-based methods, which arises from the fact that neural points are distributed around the surfaces of observed objects, encoding surrounding scene information within a fixed-radius sphere. Moreover, the relative pose computation in the loop closure relies on the existing rendering-based optimization, which may be inaccurate for large viewpoint changes thus leading to drifting of map fusion. Hence, it is interesting to design a lightweight system with a coarse-to-fine pose estimation for future work.

Acknowledgment: This work was partially supported by the NSFC (No. 62102356).

References

Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5297-5307, 2016.

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5855-5864, 2021.

Michael Bloesch, Sammy Omari, Marco Hutter, and Roland Siegwart. Robust visual inertial odometry using a direct ekf-based approach. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 298-304. IEEE, 2015.

Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan D Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics , 37(6):1874-1890, 2021.

Rohan Chabra, Jan E Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16 , pages 608-625. Springer, 2020.

Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5939-5948, 2019.

Chi-Ming Chung, Yang-Che Tseng, Ya-Ching Hsu, Xiang-Qian Shi, Yun-Hung Hua, Jia-Fong Yeh, Wen-Chin Chen, Yi-Ting Chen, and Winston H Hsu. Orbeez-slam: A real-time monocular visual slam with orb features and nerf-realized mapping. arXiv preprint arXiv:2209.13274 , 2022.

Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part II 13 , pages 834-849. Springer, 2014.

Michael Grupp. evo: Python package for the evaluation of odometry and slam. https: //github.com/MichaelGrupp/evo , 2017.

Xiyue Guo, Junjie Hu, Hujun Bao, and Guofeng Zhang. Descriptor distillation for efficient multi-robot slam. arXiv preprint arXiv:2303.08420 , 2023.

Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6001-6010, 2020.

M. M. Johari, C. Carta, and F. Fleuret. ESLAM: Efficient dense slam system based on hybrid representation of signed distance fields. In Proceedings of the IEEE international conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Mohammad Mahdi Johari, Camilla Carta, and François Fleuret. Eslam: Efficient dense slam system based on hybrid representation of signed distance fields. arXiv preprint arXiv:2211.11704 , 2022.

Marco Karrer, Patrik Schmuck, and Margarita Chli. Cvi-slam-collaborative visual-inertial slam. IEEE Robotics and Automation Letters , 3(4):2762-2769, 2018.

Georg Klein and David Murray. Parallel tracking and mapping on a camera phone. In 2009 8th IEEE International Symposium on Mixed and Augmented Reality , pages 83-86. IEEE, 2009.

Xin Kong, Shikun Liu, Marwan Taher, and Andrew J Davison. vmap: Vectorised object mapping for neural field slam. arXiv preprint arXiv:2302.01838 , 2023.

Pierre-Yves Lajoie and Giovanni Beltrame. Swarm-SLAM : Sparse Decentralized Collaborative Simultaneous Localization and Mapping Framework for Multi-Robot Systems. 2023.

Pierre-Yves Lajoie, Benjamin Ramtoula, Yun Chang, Luca Carlone, and Giovanni Beltrame. Door-slam: Distributed, online, and outlier resilient slam for robotic teams. IEEE Robotics and Automation Letters , 5(2):1656-1663, 2020.

Stefan Leutenegger, Simon Lynen, Michael Bosse, Roland Siegwart, and Paul Furgale. Keyframe-based visual-inertial odometry using nonlinear optimization. The International Journal of Robotics Research , 34(3):314-334, 2015.

Fu Li, Shaowu Yang, Xiaodong Yi, and Xuejun Yang. Corb-slam: a collaborative visual slam system for multiple robots. In Collaborative Computing: Networking, Applications and Worksharing: 13th International Conference, CollaborateCom 2017, Edinburgh, UK, December 11-13, 2017, Proceedings 13 , pages 480-490. Springer, 2018.

Hai Li, Xingrui Yang, Hongjia Zhai, Yuqian Liu, Hujun Bao, and Guofeng Zhang. Vox-surf: Voxel-based implicit surface representation. IEEE Transactions on Visualization and Computer Graphics , 2022.

Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2019-2028, 2020.

Simon Lynen, Markus W Achtelik, Stephan Weiss, Margarita Chli, and Roland Siegwart. A robust and modular multi-sensor fusion approach applied to mav navigation. In 2013 IEEE/RSJ international conference on intelligent robots and systems , pages 3923-3929. IEEE, 2013.

Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7210-7219, 2021.

Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16190-16199, 2022.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Anastasios I Mourikis and Stergios I Roumeliotis. A multi-state constraint kalman filter for vision-aided inertial navigation. In Proceedings 2007 IEEE international conference on robotics and automation , pages 3565-3572. IEEE, 2007.

Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics , 31(5):1147-1163, 2015.

Raul Mur-Artal and Juan D Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics , 33(5):1255-1262, 2017.

Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 523-540. Springer, 2020.

Tong Qin, Peiliang Li, and Shaojie Shen. Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics , 34(4):1004-1020, 2018.

Antoni Rosinol, John J Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. arXiv preprint arXiv:2210.13641 , 2022.

Patrik Schmuck and Margarita Chli. Ccm-slam: Robust and efficient centralized collaborative monocular simultaneous localization and mapping for robotic teams. Journal of Field Robotics , 36(4):763-781, 2019.

Patrik Schmuck, Thomas Ziegler, Marco Karrer, Jonathan Perraudin, and Margarita Chli. Covins: Visual-inertial slam for centralized collaboration. In 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) , pages 171-176. IEEE, 2021.

Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 , 2019.

Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison. imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6229-6238, 2021.

Zachary Teed and Jia Deng. DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. Advances in neural information processing systems , 2021.

Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5481-5490. IEEE, 2022.

Hengyi Wang, Jingwen Wang, and Lourdes Agapito. Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13293-13302, 2023.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 , 2021.

Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 , 2021.

Liu Xinyang, Li Yijin, Teng Yanbin, Bao Hujun, Zhang Guofeng, Zhang Yinda, and Cui Zhaopeng. Multi-modal neural radiance field for monocular dense slam with a light-weight tof sensor. In International Conference on Computer Vision (ICCV) , 2023.

Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5438-5448, 2022.

Xingrui Yang, Hai Li, Hongjia Zhai, Yuhang Ming, Yuqian Liu, and Guofeng Zhang. Voxfusion: Dense tracking and mapping with voxel-based neural implicit representation. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) , pages 499-507. IEEE, 2022.

Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems , 34:4805-4815, 2021.

Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems , 33:2492-2502, 2020.

Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung-Yi Lin. inerf: Inverting neural radiance fields for pose estimation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1323-1330. IEEE, 2021.

Li Yijin, Liu Xinyang, Dong Wenqi, Zhou han, Bao Hujun, Zhang Guofeng, Zhang Yinda, and Cui Zhaopeng. Deltar: Depth estimation from a light-weight tof sensor and rgb image. In European Conference on Computer Vision (ECCV) , 2022.

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847 , 2018.

Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12786-12796, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a collaborative implicit neural simultaneous localization and mapping (SLAM) system for RGB-D image sequences, which is able to merge sub-maps into a global map and also handle loop-closure for neural mapper for the first time. To facilitate the capability of doing loop-closure and sub-map merging, the authors introduce a neural point-based 3D scene representation, where each point maintains a learnable neural feature for scene encoding and is associated with a keyframe. The optimization follows the traditional SLAM’s graph optimization fashion. The experimental results on diverse datasets demonstrate the superior performance of the proposed method in both camera tracking and mapping.\n\nSTRENGTHS:\n1. Novel Neural Point-based 3D Scene Representation: A neural point cloud representation for scene encoding in the proposed framework is a significant contribution. This approach allows each point to carry encoded information, improving the system's ability to represent and cross-view association for large-scale tracking and pose refinement. \n2.Comprehensive Collaborative Framework: The authors provide a collaborative SLAM system with front-end and back-end modules, addressing essential aspects such as odometry, loop detection, sub-map fusion, and global refinement. This comprehensive approach ensures that the proposed system can handle various challenges typically encountered in SLAM tasks. \n3.Distributed-to-Centralized Learning Strategy: The incorporation of a distributed-to-centralized learning strategy is an effective solution for improving consistency and cooperation in collaborative implicit SLAM. \n4.Improved Performance: The experimental results on multiple datasets demonstrate the superiority of the proposed method in both camera tracking and mapping.\n\nWEAKNESSES:\n1.In the Introduction: This paper emphasizes the significance of targeting collaborative-SLAM, considering the substantial value already provided by loop-closure capability in effectively reducing pose drift. However, the evaluation of the paper becomes more challenging due to the necessity for additional comparative studies that can convincingly demonstrate the advantages offered by both approaches.\n\n2.In Section 2: For collaborative mapping, this paper should cite more recent works including:\n[1*] Campos, Carlos, et al. \"Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam.\" IEEE Transactions on Robotics 37.6 (2021): 1874-1890.\n[2*] Tian, Yulun, et al. \"Kimera-multi: Robust, distributed, dense metric-semantic slam for multi-robot systems.\" IEEE Transactions on Robotics 38.4 (2022).\n\nIt is also expected to provide comparative studies to the above researches.\n\n3.In Section 3: \n(1). Line 101: This paper introduced 4x4 patches to compute the point features. However, it misses details on there is a limitation on the number of patches. What if we increase the size of the patches? Will it impact the later performance? \n(2) This paper introduced so many symbols but without giving any explanation of the symbols, which makes it really hard to follow the paper. Please revise the whole paper and make them clear to the readers.\n(3) Equation 10: The explanation “Considering the strong non-convexity of the color map, only the geometry part is included in tracking loss” for not using color in the loss is not convincing. The authors are expected to provide ablation studies to prove the claim.\n\n4.In Section 4: More experiments are expected:\n(1). More comparative studies on Collaborative-SLAM are expected.\n(2). For single-agent cases: The local covisibility map can be formed as well; it would be interesting to only look at cases without a loop but having a local covisibility map.\n(3). Ablation study on “Distributed-to-Centralized Learning” is needed to prove why this is needed.\n\nQUESTIONS:\n1.In the context of this paper, which relies on point features for establishing frame-to-frame correspondence, an important concern arises regarding the maintenance of continuous tracking. Unlike the traditional SLAM pipeline, where feature locations typically reside around corners, this paper selects points without necessarily adhering to this characteristic. Thus, it becomes crucial to address how this approach ensures the continuity of tracking throughout the SLAM process.\n2.Collaborative mapping can be done by recent work on No Pose Nerf:\n[]Bian, Wenjing, et al. \"Nope-nerf: Optimising neural radiance field with no pose prior.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\nThis paper should provide a comparison to show the advantage of the proposed method.\n3. This paper has too many contributions, which does not form a scientific paper but more an engineering paper. Not sure if this paper fall into the NeuralIPS group.\n\nLIMITATIONS:\n1. The writing should be improved, especially the equations should be better explained.\n2. More experiments and ablation studies should be provided to demonstrate the advantage of the proposed algorithms. \n3. The literature review needs to refer to more recent research.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a collaborative RGB-D neural implicit SLAM method named CP-SLAM. CP-SLAM is based on neural point representation similar to PointNeRF, which is natually suitable for collaborative SLAM and loop closing. Based on this neural point representation, CP-SLAM is able to perform full SLAM with a single or multiple agents: tracking, mapping, loop detection, loop closing, submap merging, etc. Experiments show that CP-SLAM achieves SOTA results compared with previous collaborative SLAM and neural implicit SLAM methods.\n\nSTRENGTHS:\n1. Using neural point representation in the context of neural implicit SLAM is novel and the motivation of using it is also clear, i.e. more suitable for loop closing and sub-map merging.\n\n2. A full SLAM system that is capable of loop closing, Pose Graph Optimisation and sub-map merging. This is the first time I saw a full SLAM system in the context of neural implicit SLAM.\n\n3. The system seems well engineered, putting many components together and works effectively well.\n\n4. Competitive tracking and mapping results compared with previous collaborative and neural SLAM method.\n\nWEAKNESSES:\nThe motivation is clear and the method is very well engineered, but from novelty perspective, all the components in CP-SLAM are already there with some incremental modification: e.g. the scene representation is from PointNeRF, Loop detection is from an off-the-shelf NetVLAD model, etc.\n\nQUESTIONS:\n1. From Sec. 3.1 it seems that at each frame a new set of neural points are created densely via back-projection, and there doesn't seem to be any point merging scheme. Then how is CP-SLAM able to achieve 0.62MB parameters size in Tab. 4?\n\n2. Also from Sec. 3.1 it seems that the rendering loss will only optimize the neural features stored in the points but will never affect the locations of the points, which means once created the location of a point will never change unless a loop is detected and a Pose Graph Optimisation is performed. If this is the case, then the entire mapping process would be equivalent to simply back-projecting the depth images?\n\n3. How did you convert your point-based map to mesh for evaluation in Tab. 3?\n\n4. Following my question in 2: if CP-SLAM's map points are obtained explicitly from depth measurement, then it is supposed to have better accuracy than NICE-SLAM and Vox-Fusion (the same reason that TSDF-Fusion has better accuracy than neural SLAM methods). Thus, better accuracy numbers in Tab. 3 is not enough to prove CP-SLAM have \"more powerful geometric reconstruction capability\". Also it's not clear why the authors didn't report completion.\n\nAfter rebuttal: the authors have addressed all my questions and concerns.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel collaborative RGB-D SLAM pipeline, where the map is represented by an implicit field based on Point-NERF that are transformed together with their keyframes allowing pose graph optimization. They use a loop closure detection based on learned features (NetVLAD) also for sub-map alignment. They propose a central federated learning based averaging and refinement of the distributed MLPs after submap alignment.\n\nSTRENGTHS:\nThe use of point based features nicely connects the generalization and compression of implicit mapping with the ability to do efficient map refinement. The results on tracking and reconstruction show a large improvement compared to existing works.\n\nWEAKNESSES:\nThe explanations are sometimes a bit vague. E.g. line 101 what does it mean to incrementally project a pixel to a 3d location?\nThe initialization of system is not described well.\nWhile the authors describe the averaging and finetuning of the MLP weights, but do not discuss if and how the point features are shared between agents.\nEvaluation was only done on synthetic data.\n\nQUESTIONS:\nWhy is there another local finetuning and weight averaging done after the central averaging and finetuning?\nHow is the finetuning done after pose graph optimization? Is it the same as for mapping e.g. eq 9?\nWould it not be possible to run ORB-SLAM3 (https://arxiv.org/pdf/2007.11898.pdf) in a multi-map setting using depth from RGB-D as a baseline?\nWhy not evaluate on real data?\nHow is the TSDF fusion done to generate the final mesh?\nWhat is the average baseline between frames in the dataset?\nHow long are the trajectories?\n\nAfter rebuttal: The authors addressed all of my questions. Thank you.\n\nLIMITATIONS:\nOne of the main limitations of framerate/view point change is addressed by the authors.. an initial experiment to showcase the effect would have been nice though.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

comments: SUMMARY:
The paper introduces a neural implicit representation based collaborative SLAM technique. They introduce a learnable feature encoding for each point in the 3d scene based on neural rendering subsequently using it for depth and color map estimation. They introduce a centralized learning strategy to share MLPs and learn better weights. They also introduce a LM algorithm based global pose optimization and key frame centric point cloud adjustment.

STRENGTHS:
The federated learning strategy improves the neural rendering performance. Map refinement based on keyframe centric neural point field.
By encoding scene geometry and appearance with 3D points, when camera poses are optimized, these features are easily adjusted as well.

WEAKNESSES:
1. The paper only evaluates on one synthetic datasets. It would be nice to see evaluations on more datasets 
2. Better and more robust neural rendering techniques could have been used to improve the radiance, depth and color rendering

QUESTIONS:
1. The paper is not clear whether it includes neural field generation time in tracking/mapping time while comparing with Nice SLAM. 
2. The paper lacks citations. For example, in the mapping and tracking section, a similar technique is used in Nice SLAM and I see no mention of it.

LIMITATIONS:
1.The paper does produce SOTA results on some of the scenes of the dataset chosen. No explanation is given to why it doesn't produce better results on the rest. 
2. The paper is not novel enough. It rather combines the best of many other works and produces good results.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents CP-SLAM, a collaborative neural implicit SLAM system developed by the authors. CP-SLAM incorporates neural point-based odometry, loop detection, sub-map fusion, and global refinement. Additionally, the authors propose a novel neural point 3D scene representation with keyframes to enhance map fusion and adjustment. They introduce innovative learning and optimization frameworks to ensure accurate and consistent 3D mapping for cooperative localization and mapping. The performance of CP-SLAM is evaluated on various indoor RGB-D sequences, showcasing its superior mapping and camera tracking capabilities. The contributions made by the authors significantly advance the field of SLAM and offer state-of-the-art solutions.\n\nSTRENGTHS:\n1. CP-SLAM introduces a novel neural point-based representation, enabling dense collaborative neural SLAM.\n2. CP-SLAM retains the front-end and back-end modules like traditional SLAM systems, providing a comprehensive pipeline.\n3. CP-SLAM outperforms state-of-the-art methods in both localization and reconstruction. The CP-SLAM system supports both single-agent and multi-agent modes, allowing for versatile evaluation.\n4. The system is evaluated using datasets based on the Replica scenes, providing a realistic and representative testing environment.\nCP-SLAM is compared against recent neural implicit RGB-D SLAM methods in single-agent experiments, showcasing its performance in loop closure.\n5. In the two-agent experiments, CP-SLAM is compared with a traditional method, highlighting its collaborative capabilities.\n6. The ablation studies are conducted to demonstrate the importance of different modules in the proposed system.\n7. The implementation details, such as GPU specifications and parameter settings, are provided, ensuring reproducibility.\n\nWEAKNESSES:\n1. CP-SLAM requires significant GPU resources to handle multiple image sequences. The relative pose computation in loop closure relies on existing rendering-based optimization, which may be inaccurate for large viewpoint changes and lead to map fusion drifting.\n2. The comparison with traditional methods in the two-agent experiments may not fully represent the performance of other collaborative neural SLAM approaches.\n3. CP-SLAM's performance is limited by its GPU resource demand, making it less suitable for resource-constrained devices or environments.\n4. Currently, a large-scale experiment is not conducted, but for multi-agents, this is an important point to consider.\n\nQUESTIONS:\n1. How does CP-SLAM handle scalability in terms of the size and complexity of the datasets?\n2. Can CP-SLAM be applied to real-world datasets to evaluate its performance in practical scenarios?\n3. Are there any specific limitations or challenges in the implementation of CP-SLAM that need to be addressed?\n4. Can Nicer-SLAM and DROID-SLAM be compared as a contrast?\n5. The article mentioned \"Naturally, it is expected that the neural point cloud layout should be rearranged following global pose graph optimization. However, a world-centric point cloud map obviously cannot allow such adjustment. To tackle this limitation, we propose a keyframe-centric neural point field, where each 3D point is associated with a keyframe\". Can the authors better explain why a world-centric point cloud map obviously cannot allow such adjustment? \n\nSincerely,\n\nLIMITATIONS:\nThe limitations have been discussed in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThere have been some papers on getting dense Visual SLAM with neural fields published in the last couple of years. In this work, the authors proposed a set of additions to prior pipelines that get the NN SLAM closer to the traditional approaches concerning the overall framework. Specifically, the authors addressed loop detection, pose graph optimization, bundle adjustment, and collaborative SLAM.\n\nSTRENGTHS:\n1) Visual-SLAM with neural fields has the huge benefit of providing a dense map of the environment. The topic is relevant to computer vision, robotics, and I believe it fits the NeurIPS scope.\n2) The paper is well-written, easy to follow, and well-organized. Related work is OK.\n3) Although the technical novelties are relatively limited, in my opinion, the contributions to the pipeline are significant.\n\nWEAKNESSES:\n1) As written above, the technical contributions are limited. The paper focuses more on adding new steps to prior Visual SLAM pipelines.\n2) The inclusion of the NetVLAD (not new) as a tool for loop detection looks like a trick. It is not elegant, in the sense that it is a parallel feature extractor that (maybe) could be done using the neural field encoders.\n3) Only one dataset and 4 sequences are used, which is small concerning quantity and variety.\n4) I would prefer to see more comparisons with previous traditional methods (without neural fields).\n5) Figures 5 and 6 are not easily readable.\n\nQUESTIONS:\n1) I would like the authors to explain what is the motivation for the inclusion of NetVLAD. Can't those descriptors be obtained from the neural field?\n2) It is written that a pre-trained NetVLAD was used. It would be important to give more details about this training and how it generalizes to different scenes.\n3) Usually, with classical V-SLAM, the bundle adjustment estimates both the pose of the cameras and the 3D map simultaneously. It seems that the authors are doing this alternatively. I would like to see some comments on this.\n4) Why are the results for Apartment 2 so bad for the proposed approach? I would like the authors to comment on this.\n5) It seems that traditional V-SLAM methods using RGB-D are missing (such as OrbSLAM). I would like to see some comments on this.\n6) Maybe I missed something. But it is not clear to me how is the error in Tab. 2 computed.\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}