SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CP-SLAM: Collaborative Neural Point-based SLAM
Jiarui Hu<sup>1</sup>, Mao Mao<sup>1</sup>, Hujun Bao<sup>1</sup>, Guofeng Zhang<sup>1</sup>, Zhaopeng Cui<sup>1*</sup>
$^{1}$ State Key Lab of CAD&CG, Zhejiang University
This paper presents a collaborative implicit neural simultaneous localization and mapping (SLAM) system with RGB-D image sequences, which consists of complete front-end and back-end modules including odometry, loop detection, sub-map fusion, and global refinement. In order to enable all these modules in a unified framework, we propose a novel neural point based 3D scene representation in which each point maintains a learnable neural feature for scene encoding and is associated with a certain keyframe. Moreover, a distributed-to-centralized learning strategy is proposed for the collaborative implicit SLAM to improve consistency and cooperation. A novel global optimization framework is also proposed to improve the system accuracy like traditional bundle adjustment. Experiments on various datasets demonstrate the superiority of the proposed method in both camera tracking and mapping.
Dense visual Simultaneous Localization and Mapping (SLAM) is an enduring fundamental challenge in computer vision, which aims to achieve 3D perception and exploration of unknown environments by self-localization and scene mapping with wide downstream applications in autonomous driving, unmanned aerial vehicle(UAV) navigation, and virtual/augmented reality (VR/AR).
The traditional visual SLAM has witnessed continuous development, leading to accurate tracking and mapping in various scenes. Most of the current prominent visual SLAM systems pay their primary attention to real-time tracking performance [29, 31, 4], while dense map reconstruction is normally achieved with the temporal fusion of additional depth input or estimated depth images, thus sensitive to noises and outliers. Some collaborative visual SLAM systems [34, 20] have also been proposed as a straightforward extension of monocular visual SLAM.
The learning-based visual SLAM has attracted more attention recently with better robustness against noises and outliers [37]. Very recently, some methods [36, 50, 44] exploit the Neural Radiance Fields (NeRF) for dense visual SLAM in a rendering-based optimization framework showing appealing rendering quality in novel view. However, different from the traditional feature-based SLAM system, such learning-based methods with implicit representation are normally pure visual odometry systems without loop closure and pose graph optimization due to the limitation of the scene representation (e.g., a neural network or feature grid), which also makes it hard to be adapted to the collaborative SLAM. Take the feature grid representation as an example, it is hard to adjust or align the feature grid when the pose is optimized after loop closure or transformed to the unified global coordinate system for collaborative SLAM.
In this paper, we introduce a novel collaborative neural point-based SLAM system, named CP-SLAM, which enables cooperative localization and mapping for multiple agents and inherently supports loop closure for a single agent. However, it is nontrivial to design such a system. At first, we need a new neural representation for SLAM that is easy to be adjusted for loop closure and collaborative SLAM. Inspired by Point-NeRF [43], we built a novel neural point-based scene representation with keyframes.
The scene geometry and appearance are encoded in 3D points with per-point neural features, and each point is associated with a certain keyframe. In this way, when the camera poses are optimized with loop closure and pose graph optimization, these neural points can be easily adjusted like traditional 3D points. Second, different from the monocular implicit SLAM system, a new learning strategy is required for collaborative implicit SLAM. For better consistency and cooperation, we present a two-stage learning strategy, i.e., distributed-to-centralized learning. In the first stage (before sub-map fusion), we set up an independent group of decoders for each RGB-D sequence and update them separately. In the second stage (after sub-map fusion), we fuse weights from all groups of decoders and continue with lightweight fine-tuning, after which all sequences can share and jointly optimize a common group of decoders. Furthermore, like the bundle adjustment in traditional SLAM, a novel optimization framework is needed to adjust both the camera poses and scene geometry for the neural implicit SLAM system. To this end, we introduce the pose graph optimization into the implicit SLAM system followed by a global map refinement.
Our contributions can be summarized as follows. At first, we present the first collaborative neural implicit SLAM system, i.e., CP-SLAM, which is composed of neural point based odometry, loop detection, sub-map fusion, and global refinement. Second, we propose a new neural point 3D scene representation with keyframes, which facilitates map fusion and adjustment. Furthermore, novel learning and optimization frameworks are proposed to ensure consistent and accurate 3D mapping for cooperative localization and mapping. We evaluate CP-SLAM on a variety of indoor RGB-D sequences and demonstrate state-of-the-art performance in both mapping and camera tracking.
Single-agent Visual SLAM. With the development of unmanned intelligence, visual SLAM becomes an active field in the last decades. Klein et al. proposed a visual SLAM framework [15] that separates tracking and mapping into different threads. This framework is followed by most of current methods. Traditional single-agent visual SLAM uses filtering or nonlinear optimization to estimate the camera pose. Filtering based methods [27, 3, 23] are more real-time capable but less globally consistent. In contrast, optimization-based [28, 29, 31, 4, 8, 19] methods can make full use of past information. Meanwhile, multi-source information is widely incorporated into the SLAM system to improve specific modules, such as light-weight depth estimation [48, 42].
Collaborative Visual SLAM. Collaborative SLAM can be divided into two categories: centralized and distributed. CVI-SLAM [14], a centralized visual-inertial framework, can share all information in a central server and each agent outsources computationally expensive tasks. In centralized SLAM, the server manages all sub-maps, performs map fusion and global bundle adjustment, and feeds processed information back to each agent. This pipeline is reproduced in CCM-SLAM [33], where each agent is equipped with a simple visual odometry and sends localization and 3D point cloud to the central server. In terms of distributed systems, Lajoie et al. developed a fully distributed SLAM [18], which is based on peer-to-peer communication and can reject outliers for robustness. NetVLAD [1] is used in [18] to detect loop closures. In addition, [10] proposed the compact binary descriptor specifically for multi-agent system. Compared with traditional collaborative systems, our system can perform dense mapping with fewer neural points.
Neural Implicit Representation. Neural implicit field showed outstanding results in many computer vision tasks, such as novel view synthesis [24, 25, 38, 2], scene completion [5, 11, 30] and object modelling [6, 22, 40, 45, 46]. In recent study, some works [47, 41] attempted to reversely infer camera extrinsics from the built neural implicit field. Inspired by different representations of neural field including voxel grid [21] and point cloud [43], NICE-SLAM [50] and Vox-Fusion [44] chose voxel grid to perform tracking and mapping instead of a single neural network which is limited by expression ability and forgetting problem. Both of them are most related works to ours. Besides, vMAP [16] can efficiently model watertight object models in the absence of 3D priors. ESLAM [13] turns to tri-plane and Truncated Signed Distance Field (TSDF) to solve RGB-D SLAM. In addition to purely neural-based methods, some hybrid systems, such as NeRF-SLAM [32] and Orbeez-SLAM [7], construct the neural map but use traditional methods to estimate poses.
The overview of our collaborative SLAM system is shown in Fig. 1. Given a set of RGB-D sequences, our system incrementally performs tracking and mapping based on neural point cloud representation for each agent (Section 3.1). We incorporate a learning-based loop detection module that extracts unique descriptors for 2D frames, and stitches sub-maps through high-quality loop constraints
Figure 1: System Overview. Our system takes single or multi RGB-D streams as input and performs tracking and mapping as follows. From left to right, we conduct differentiable ray marching in a neural point field to predict depth and color. To obtain feature embedding of a sample point along a ray, we interpolate neighbor features within a sphere with radius  $r$ . MLPs decode these feature embeddings into meaningful density and radiance for volume rendering. By computing rendering difference loss, camera motion and neural field can be optimized. While tracking and mapping, a single agent continuously sends keyframe descriptors encoded by NetVLAD to the descriptor pool. The central server will fuse sub-maps and perform global pose graph optimization(PGO) based on matching pairs to deepen collaboration. Finally, our system ends the workflow with keyframe-centric map refinement.
(Section 3.2). We further design a two-stage (distributed-to-centralized) MLP training strategy to improve consistency and strengthen collaboration (Section 3.3). To reduce cumulative error in mapping and tracking, we use co-visibility among sub-maps to take global pose graph optimization as the back-end processing, followed by frame-based map refinement (Section 3.4). We will elaborate on the entire pipeline of our system in the following subsections.
3.1 Neural Point based Odometry
Our system starts from a front-end visual odometry module, in which we combine pose backpropagation update and point-based neural field to perform sequential tracking and mapping.
Point-based Neural Field. We divide an image into  $4 \times 4$  patches and incrementally project each central pixel of every patch to its corresponding 3D location  $p \in \mathbb{R}^3$ . To acquire the feature embedding  $f \in \mathbb{R}^{32}$  anchored on  $p$ , a 2D image is fed into an untrained single-layer convolutional neural network. We define our neural point cloud as:
$$
P = \left\{p _ {j}, f _ {j} \mid j = 1, \dots , N \right\}, \tag {1}
$$
Volume Rendering. We emulate the differentiable volume rendering strategy in NeRF [26], where radiance and occupancy are integrated along rays to render color and depth maps. Given a 6DoF pose  $\{\mathbf{R_c},\mathbf{O_c}\}$  and intrinsics of a frame, we can cast rays from randomly selected pixels and sample  $N_{total}$  points. Any sampling point is defined as
$$
x _ {i} = \mathbf {O} _ {\mathbf {c}} + t _ {i} * \mathbf {d} i \in 1 \dots N _ {\text {t o t a l}}, \tag {2}
$$
where  $t_i \in \mathbb{R}$  is the depth of the point and  $\mathbf{d} \in \mathbb{R}^3$  is the unit ray direction. Guided by depth information, we can distribute our samples near the real surface. Specifically, we sample  $N_{total} = N_{near} + N_{uni}$  points on each ray. For any pixel with valid depth  $D$ ,  $t_i$  is uniformly sampled within the intervals [0.95D, 1.05D] and [0.95Dmin, 1.05Dmax] respectively, yielding  $N_{near}$  and  $N_{uni}$  sample points correspondingly, where  $D_{min}$  and  $D_{max}$  are minimum and maximum depth values of the current depth map. For any pixel without valid depth values,  $t_i$  is uniformly sampled within the interval  $[D_l, 1.05D_{max}]$ , generating  $N_{total}$  points. For each point  $x_i$ , we firstly query  $K$  neighbors  $\{p_k | k = 1, \dots, K\}$  within query radius  $r$  and then use an MLP  $C$  to convert the original neighbor feature  $f_k$  into  $f_{k,x_i}$  that incorporates relative distance information, i.e.,
$$
f _ {k, x _ {i}} = C \left(f _ {k}, x _ {i} - p _ {k}\right), \tag {3}
$$
Then a radiance MLP  $U$ , decodes the RGB radiance  $c$  at location  $x_{i}$  using an interpolation feature  $f_{x_i}$  which is obtained from weighted interpolation according to inverse distance weights and neighboring features:
$$
f _ {x _ {i}} = \sum_ {k} \frac {w _ {k}}{\sum w _ {k}} f _ {k, x _ {i}}, w _ {k} = \frac {1}{\left\| p _ {k} - x _ {i} \right\|}, \tag {4}
$$
$$
c = U \left(f _ {x _ {i}}\right), \tag {5}
$$
If no neighbors are found, the occupancy  $\sigma$  at  $x_{i}$  is set to zero. Otherwise, we regress  $\sigma$  using an occupancy MLP  $G$  at  $x_{i}$ . We follow a similar inverse distance-based weighting interpolation as radiance, instead of interpolating at feature level. We decode  $\sigma_{i}$  for each neighbor and finally interpolate them:
$$
\sigma_ {k} = G \left(f _ {k, x _ {i}}\right), \tag {6}
$$
$$
\sigma = \sum_ {k} \frac {w _ {k}}{\sum w _ {k}} \sigma_ {k}, \tag {7}
$$
Next we use  $c_{x_i}, \sigma_{x_i}$  regressed from MLPs to estimate per-point weight  $\alpha_{x_i}$ .  $\alpha_{x_i}$  is regarded as the opacity at  $x_i$ , or the probability of a ray terminates at this point and  $z_{x_i}$  is the depth of point  $x_i$ . Depth map and color map can be rendered by calculating depth and radiance expectations along rays as in Eq. 8.
$$
\hat {D} = \sum_ {i = 1} ^ {N _ {\text {t o t a l}}} \alpha_ {x _ {i}} z _ {x _ {i}}, \quad \hat {I} = \sum_ {i = 1} ^ {N _ {\text {t o t a l}}} \alpha_ {x _ {i}} c _ {x _ {i}}, \tag {8}
$$
Mapping and Tracking. We use rendering difference loss as described in Eq. 9 that consists of geometric loss and photometric loss during the mapping process. For a new-coming frame, we sample  $M_{1}$  pixels to optimize point-anchored features  $f_{i}$  and parameters of MLP  $C, U, G$ . With the first frame, we need to perform a good initialization at a few higher cost of  $M_{3}$  pixels and around  $3000 \sim 5000$  optimization steps to ensure smooth following processing. For subsequent mapping, we select pixels uniformly from the current frame and 5 co-visible keyframes. We find that joint mapping can effectively stabilize the tracking process.
$$
\mathcal {L} _ {\text {m a p p i n g}} = \frac {1}{M _ {1}} \sum_ {m = 1} ^ {M _ {1}} \left| D _ {m} - \hat {D} _ {m} \right| + \lambda_ {1} \left| I _ {m} - \hat {I} _ {m} \right|, \tag {9}
$$
where  $D_{m}$ ,  $I_{m}$  represent ground truth depth and color map,  $\hat{D}_{m}$ ,  $\hat{I}_{m}$  are corresponding rendering results, and  $\lambda_{1}$  is the loss balance weight. During the tracking process, we backpropagate farther to optimize camera extrinsics  $\{q,t\}$  while keeping features and MLPs fixed, where  $q$  is quaternion rotation and  $t$  is the translation vector. We sample  $M_{2}$  pixels across a new-coming frame and assume the zero motion model where the initial pose of a new frame is identical to that of the last frame. Considering the strong non-convexity of the color map, only the geometry part is included in tracking loss:
$$
\mathcal {L} _ {\text {t r a c k i n g}} = \frac {1}{M _ {2}} \sum_ {m = 1} ^ {M _ {2}} \left| D _ {m} - \hat {D} _ {m} \right|, \tag {10}
$$
3.2 Loop Detection and Sub-Map Alignment
To align sub-maps and reduce accumulated pose drift, we perform loop detection and calculate relative poses between different sub-maps. For each keyframe in the sub-map, we associate it with a descriptor generated by the pre-trained NetVLAD [1] model. We use cosine similarity between descriptors as the judgment criteria for loop detection. When the camera moves too much between frames, pose optimization tends to fall into local optimum, so we use two similarity thresholds  $\lambda_{fine}, \lambda_{coarse}$  to find matching pairs that have enough overlap and ensure correct loop relative pose. Given two sub-maps,  $\mathcal{M}_1$  and  $\mathcal{M}_2$ , we find out loop frames with largest overlap,  $\{I^{l1}, I^{l2}\}$ , with similarity greater than  $\lambda_{fine}$ , and small overlap loop frames,  $\{(I_1^{s1}, I_2^{s2}), \dots, (I_n^{s1}, I_n^{s2})\}$ , with similarity greater than  $\lambda_{coarse}$  but less than  $\lambda_{fine}$ . For pairs  $\{I^{l1}, I^{l2}\}$ , we take pose of  $I^{l1}$  as initial pose and  $I^{l2}$  as the reference frame. The tracking method described in 3.1 can be used to obtain the relative pose measurement between  $\{\mathcal{M}_1, \mathcal{M}_2\}$ . We can use this relative pose to perform a 3D
Figure 2: Voxel Grid Limitation. (a) If the initial coordinate system is not uniform, the neural field representations of existing neural SLAM such as voxel grid will suffer from misalignment during sub-map fusion. (b) The point cloud is not restricted by 3D geometry shape and can be fused freely. After sub-map fusion, We can query neighbors at  $x_{i}$  from observations of Agent1 (green point cloud) and Agent2 (purple point cloud).
rigid transformation on sub-maps, so that all sub-maps are set in the same global coordinate system, achieving the purpose of sub-map alignment as shown in Fig. 2.
$$
\mathcal {L} _ {\text {l o o p}} \left(T _ {r}\right) = \frac {1}{M _ {2}} \sum_ {\left\{I ^ {l 1}, I ^ {l 2} \right\}} ^ {M _ {2}} \left| D _ {m} \left(T _ {r}\right) - \hat {D} _ {m} \right|, \tag {11}
$$
where  $T_{r}$  is the relative pose between  $\mathcal{M}_1$  and  $\mathcal{M}_2$ , and  $D, \hat{D}$  is the rendered depth and ground truth depth. For loop frames with small overlap  $\{I_i^{s1}, I_i^{s2}\}$ , we only use them to perform pose graph optimization (refer to 3.4). Sub-map fusion will lead to neural point redundancy in specific areas, which imposes a burden on computing and memory. Due to the sparsity of the neural point cloud, we adopt a grid-based filtering strategy, that is, we perform non-maximum suppression based on the distance from a neural point to the center of a  $\rho^3$  cube.
3.3 Distributed-to-Centralized Learning
To enhance consistency and cooperation, in collaborative SLAM, we adopt a two-stage MLP training strategy. At the first stage (Distributed Stage), each image sequence is considered as a discrete individual with a unique group of MLPs  $\{C_j,U_j,G_j\}$  for sequential tracking and mapping. After loop detection and sub-map fusion, we expect to share common MLPs across all se
quences (Centralized Stage). To this end, we introduce the Federated learning mechanism which trains a single network in a cooperating shared way. At the same time as sub-map fusion, we average each group of MLPs and fine-tune the averaged MLPs on all keyframes to unify discrete domains. Subsequently, we iteratively transfer sharing MLPs to each agent for local training and average the local weights as the final optimization result of sharing MLPs, as shown in Fig. 3.
Figure 3: Two-stage Learning Strategy.
3.4 Pose Graph Optimization and Global Map Refinement
After front-end processing of all sequences, including tracking, mapping, loop detection, and sub-map fusion, we establish a global pose graph model, in which per-frame poses are nodes, sequential relative pose, and loop relative pose are edges. The pose graph model is illustrated in Fig. 4. We carry out global pose graph optimization across the entire pose graph, referring to tradi
Single-agent Pose Graph
Two-agent Pose Graph
Keyframe-centric Model
Figure 4: Pose Graph and Keyframe-centric Model.
tional visual SLAM, to force the estimated trajectory closer to the ground truth. Global pose graph
optimization effectively mitigates cumulative error and improves tracking accuracy. We use the Levenberg Marquaret algorithm to solve this nonlinear global pose graph optimization problem described by Eq. 12, where  $v$  is the set of nodes,  $E_{s}$  is the set of sequential edges,  $E_{l}$  is the set of loop edges and  $\Lambda_{i}$  represents the uncertainty of corresponding edges.
$$
v ^ {*} = \arg \min  _ {v} \frac {1}{2} \sum_ {e _ {i} \in E _ {s}, E _ {l}} e _ {i} ^ {T} \Lambda_ {i} ^ {- 1} e _ {i}, \tag {12}
$$
Naturally, it is expected that the neural point cloud layout should be rearranged following global pose graph optimization. However, a world-centric point cloud map obviously cannot allow such adjustment. To tackle this limitation, we propose a keyframe-centric neural point field, where each 3D point is associated with a keyframe (Fig. 4). Benefitting from this design, we can refine our point cloud 3D locations according to optimized poses. Following global map refinement, we employ grid-based filtering to deal with neural point cloud redundancy occurring in local regions. Considering the slight mismatch between refined neural fields and decoders after global map refinement, we end up performing a low-cost fine-tuning of the global neural point fields with fewer optimization iterations.
Our CP-SLAM system supports both single-agent and multi-agent modes. Thus, we evaluate our proposed collaborative SLAM system in two aspects, both single-agent experiments with loop closure and two-agent experiments, of varying sizes and complexity. In terms of a single agent, we generate datasets based on the Replica [35] scenes and then compare our method against recent neural and traditional RGB-D SLAM methods. For the two-agent side, since no collaborative neural SLAM work has emerged so far, we compare our method with traditional methods. We also conduct ablation studies to show the importance of modules in the proposed system.
Implementation Details. CP-SLAM system runs an RGB-D sequence on an NVIDIA RTX3090 GPU. In the two-agent experiment, we need an additional RTX3090 as the central server. To encode higher frequency detail information, during inference, we impose position encoding on the origin neighbor feature  $f_{i}$  and relative distance  $\| p_i - x\|$  with an order of 1 and 7. We utilize the FRNN library to query  $K = 8$  nearest neighbors on GPU. In all our experiments, we set  $N_{near} = 16$ ,  $N_{uni} = 4$ ,  $\lambda_1 = 0.2$ ,  $D_l = 0.001m$ ,  $r = 0.15m$ ,  $\rho = 0.14m$ ,  $M_1 = 3000$ ,  $M_3 = 3136$ ,  $M_2 = 1500$ . We extract a keyframe every 50 frames and perform map optimization and point cloud supplementation every 10 frames. For single-agent experiments, we optimize the neural field for 200 iterations. For two-agent experiments, considering that features anchored on neural points do not need to be trained from scratch after sub-map fusion, we reduce the number of iteration steps to 150. Further implementation details can be found in our supplementary material.
Baselines. In the single-agent experiment, because we use the rendered loop-closure data, we primarily choose the state-of-the-art neural SLAM systems such as NICE-SLAM [50], Vox-Fusion [44] and ORB-SLAM3 [4] for comparison on the loop-closure dataset. For the two-agent experiment, we compare our method with traditional approaches, such as CCM-SLAM [33], Swarm-SLAM [17] and ORB-SLAM3 [4].
Datasets. For reconstruction assessment, we utilize the synthetic dataset Replica [35], equipped with a high-quality RGB-D rendering SDK. We generate 8 collections of RGB-D sequences, 4 of which represent single-agent trajectories, each containing 1500 RGB-D frames. The remaining 4 collections are designed for collaborative SLAM experiments. Each collection is divided into 2 portions, each holding 2500 frames, with the exception of Office-0-C which includes 1950 frames per part.
Metrics. As a dense neural implicit SLAM system, we quantitatively and qualitatively measure its mapping and tracking capabilities. For mapping, we evaluate L1 loss between 196 uniformly-sampled depth maps, which are rendered from our neural point field, and ground truth ones. Furthermore, in terms of 3D triangle mesh, we compute mesh reconstruction accuracy. For tracking, we use ATE RMSE, Mean and Median to comprehensively measure trajectory accuracy so as to prevent the negative impact caused by a few extreme outliers.
4.1 Two-agent Collaboration
We provide the quantitative results of two-agent experiments on four scenes including Replica [35] Apartment-0, Apartment-1, Apartment-2 (multi-room), and Office-0-C (single-room). We take RGB-based CCM-SLAM [33], RGBD-based Swarm-SLAM [17] and traditional ORB-SLAM3 [4] for comparison. Table. 1 reports the localization accuracy of different methods. Despite being affected