CP-SLAM: Collaborative Neural Point-based SLAM System
Authors: Jiarui Hu
Mao Mao
Hujun Bao
Guofeng Zhang
Zhaopeng Cui
Venue: NeurIPS 2023 poster (2023)
Keywords: Collaborative SLAM; Neural Point Field; Keyframe-based SLAM; Pose Graph Optimization
TL;DR: A collaborative implicit neural point-based localization and mapping (SLAM) system for monocular RGB-D image, including loop detection and pose graph optimization.

Abstract: This paper presents a collaborative implicit neural simultaneous localization and mapping (SLAM) system with RGB-D image sequences, which consists of complete front-end and back-end modules including odometry, loop detection, sub-map fusion, and global refinement. In order to enable all these modules in a unified framework, we propose a novel neural point based 3D scene representation in which each point maintains a learnable neural feature for scene encoding and is associated with a certain keyframe. Moreover, a distributed-to-centralized learning strategy is proposed for the collaborative implicit SLAM to improve consistency and cooperation. A novel global optimization framework is also proposed to improve the system accuracy like traditional bundle adjustment. Experiments on various datasets demonstrate the superiority of the proposed method in both camera tracking and mapping.

CP-SLAM: Collaborative Neural Point-based SLAM
Jiarui Hu', Mao Mao', Hujun Bao', Guofeng Zhang', Zhaopeng Cui
State Key Lab of CAD&CG, Zhejiang University
Abstract
This paper presents a collaborative implicit neural simultaneous localization and mapping (SLAM) system with RGB-D image sequences, which consists of com- plete front-end and back-end modules including odometry, loop detection, sub-map fusion, and global refinement. framework, we propose novel neural point based 3D scene representation in which each point maintains learnable neural feature for scene encoding and is associated with a certain keyframe. Moreover, a iiistributed-to-cnttalll strategy is proposed for the collaborative implicit SLAM to improve consistency and cooperation. A novel global optimization framework is also proposed to im- prove the system accuracy like traditional bundle adjustment. Experiments on various datasets demonstrate the superiority of the proposed method in both camera tracking and mapping.
order to enable all these modules in a unified
learning
1 Introduction
Dense visual Simultaneous Localization and Mapping (SLAM) is enduring fundamental challenge in computer vision, which aims to achieve 3D perception and exploration unknown environments by self-localization and scene mapping with wide downstream applications in autonomous driving,
unmanned aerial vehicle(UAV) navigation, and virtual/augmented reality (VR/AR).
The traditional visual SLAM has witnessed continuous development, leading accurate tracking and mapping in various scenes. Most of the current prominent visual SLAM systems pay their primary attention to real-time tracking performance 229 31 4 while dense map reconstruction is normally achieved with the temporal fusion of additional depth input or estimated depth images, thus sensitive to noises and outliers. Some collaborative visual SLAM systems (34 20 have also been proposed as a straightforward extension of monocular visual SLAM.
The learning-based visual SLAM has attracted more attention recently with better robustness against noises and outliers (37] Very recently, some methods 36 50 exploit the Neural Radiance Fields (NeRF) for dense visual SL.AM in rendering- optimization framework showing appealing rendering quality in novel view. However, different from the traditional feature- SLAM system, such saaning-basce methods with implicit representation without loop closure and pose graph optimization due to the limitation of the scene representation (e.g., g..& neural network or feature grid), which also makes it hard to be adapted to the collaborative SLAM. Take the feature grid representation as an example, it is hard to adjust or align the feature grid when the pose is optimized after loop closure transformed the unified global coordinate system for collaborative SLAM.
normally pure visual odometry systems
In this paper, we introduce novel collaborative neural point-based SLAM system, named CP-SLAM, which enables cooperative localization and mapping for multiple agents and inherently supports loop closure for single agent. However, it is nontrivial to design such system. At first, we need new neural representation for SLAM that is easy to be adjusted for loop closure and collaborative SLAM. Inspired by Point-NeRF (43) we built novel neural bosnt-based scene representation with keyframes.
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The scene geometry and appearance are encoded in 3D points with per-point neural features, and each point is associated with certain keyframe. In this way, when the camera poses are optimized with loop closure and pose graph optimization, these neural points can be easily adjusted like traditional 3D points. Second, different from the monocular implicit SL.AM system, a new learning strategy is required for collaborative implicit SLAM. For better consistency and cooperation, we present two-stage learning strategy, i.c., fusion), we set up independent group of decoders for each RGB-D sequence and update them separately. In the second stage (after sub-map fusion), we fuse weights from all groups of decoders and continue with lightweight fine-tuning, after which all sequences can share and jointly optimize common group of decoders. Furthermore, like the bundle adjustment in traditional SLAM, a novel optimization framework is needed to adjust both the camera poses and scene geometry for the neural implicit SL ALAM system. To this end, we introduce the pose graph optimization into the implicit SLAM system followed by a global map refinement.
learning. In the first stage (before sub-map
Our contributions can be summarized as follows. first, we present the first collaborative neural implicit SLAM system, i.e., CP-SLAM, which is composed of neural point based odometry, loop detection, sub-map fusion, and global refinement. Second, we propose a new neural point 3D scene representation with keyframes, which facilitates map fusion and adjustment. Furthermore, novel learning and optimization frameworks are proposed to ensure consistent and accurate 3D mapping for cooperative localization and mapping. We evaluate CP-SLAM on variety of indoor RGB-D sequences and demonstrate offtc-of-th-aatt performance in both mapping and camera tracking.
2 Related Work
Single-agent Visual SLAM. With the development of unmanned intelligence, visual SLAM becomes an active field in the last decades. Klein al. proposed visual SLAMM framework [15] that separates tracking and mapping into different threads. This framework is followed by most of current methods. Traditional single-agent visual SLAM uses filtering or nonlinear optimization to estimate the camera are more real-time capable but less globally consistent. In
pose. Filtering based methods [27
3 23
contrast, optimization- Mcanwhile, multi-source information is widely incorporated into the SLAM system to improve specific modules. such as light-weight depth estimation [48] (42..
28
8 [9] methods can make full use of past information.
Collaborative Visual SLAM. Collaborative SLAM can be divided into two categories: centralized and distributed. CVI-SLAM [14,, a centralized visual-inertial framework, can share all information in central server and each agent outsources computationally expensive tasks. In centralized SLAM, the server manages all sub-maps, performs map fusion and global bundle adjustment, and feeds processed information back each agent. This pipeline is reproduced in CCM-SLAM [X]] where each agent is equipped with a simple visual odometry and sends localization and 3D point cloud to the central server. In terms of distributed systems, Lajoie et al. developed fully distributed SLAM (18) which is based on peer-to-peer communication and can reject outliers for robustness. LetD II] used in [18] to detect loop closures. In addition. [00] proposed the compact binary descriptor specifically for multi-agent system. Compared with traditional collaborative systems, our system can perform dense mapping with fewer neural points.
Neural Implicit Representation. Neural implicit field showed outstanding results in many computer vision tasks, such as novel view synthesis (4 25
38 2 scene completion 5 30 and object
modelling [6] 40
45 46
In recent study, some works [47 41 attempted to reversely infer
camera extrinsics from the built neural implicit field. Inspired by different representations of neural field including voxel grid (2I) and point cloud 43 NICE-SLAM (50) and Vox-Fusion [44) chose voxel grid to perform tracking and mapping instead of a single neural network which is limited by expression ability and forgetting problem. Both of them most related works to ours. Besides, vMAP [6] can efficiently model watertight object models in the absence of 3D priors, ESLAM [3] turns to tri-plane and Truncated Signed Distance Field (TSDF) to solve RGB-D SLAM. In addition to purely neural-based methods, some hybrid systems, such as NeRF-SLAM 32 and Orbeez-SLAM (7]. construct the neural map but use traditional methods to estimate poses.
3 Method
The overview of our collaborative SLAM system is shown Fig, [ Given a set of RGB-D sequences, our system incrementally performs tracking and mapping based on neural point cloud representation for each agent (Section (3.Tp. We incorporate learning-based loop detection module that extracts unique descriptors for 2D frames, and stitches sub-maps through high-quality loop constraints
2

---

Color Loss
Aauppis
Ayswsing
Sequence-1
Dosth Loss
P.R
PGO
A p-
NetVLAD
Ayine
Assars
Loop Bdlges
Keyframe-centee
Map Refinement
Descriptor Pool
Sub-map Fusion
Similiar Process as Sequence-1
Sequence-2 2
Figure 1: System Overview. Our system takes single multi RGB-D streams as input and performs tracking and mapping as follows. From left to right, we conduct differentiable ray marching in a neural point field to predict depth and color. To obtain feature embedding of a sample point along a ray, we interpolate neighbor features within a sphere with radius r. MLPs decode these feature embeddings into meaningful density and radiance for volume rendering. By computing rendering difference loss, camera motion and neural field can be optimized. While tracking and mapping, single agent continuously sends keyframe descriptors encoded by NetVLAD to the descriptor pool. The central server will fuse sub-maps and perform global pose graph optimization based on matching pairs to deepen collaboration. Finally, our system ends the workflow with keyframe- centric map refinement.
(Section 3.2p. We further design two-stage improve consistency and strengthen collaboration (Section (.3). To reduce cumulative error in mapping and tracking, we use co-visibility among sub-maps to take global pose graph optimization as the back-end processing, followed by frame- based map refinement (Section 3.4 We will elaborate on the entire pipeline of our system in the following subsections.
caapaauudd--t--nnttliiieeei MLP training strategy to
3.1 Neural Point based Odometry
Our system starts from front-end visual odometry module, in which we combine pose back- propagation update and point-based neural field to perform sequential tracking and mapping. Point-based Neural Field. We divide image into 4 patches and incrementally project each central pixel of every patch to its corresponding 3D location E R To acquire the feature embedding 32 anchored on p. a 2D image is fed into an untrained single-layer convolutional neural network. We define our neural point cloud as:
P= (p = I,...,
Volume Rendering. We emulate the differentiable volume rendering strategy in NeRF (26). where radiance and occupancy are integrated along rays to render color and depth maps. Given a 6DoF pose (Re; Oa} and intrinsics of a frame, can cast rays from randomly selected pixels and sample Neotal points. Any sampling point is defined as
2; O t, + d E 1...Ntotal,
(2)
where IR is the depth of the point and d R3 is the unit ray direction. Guided by depth information, we can distribute our samples near the real surface. Specifically, we sample Ntotal = Nnear Nuni points each ray. For any pixel with valid depth D. l, is uniformly sampled within respectively, yielding Nnear and Nuni sample points correspondingly. where Dmin and Dman are minimum and maximum depth values of the current depth map. For any pixel without valid depth values, &; is uniformly sampled within the interval [Di, 1.055mmaal generating Nrotal points. For each point z; firstly query K neighbors (pulk k 1, ..., K} within query radius and then use an MLP C' to convert the original neighbor feature fa into fk.z. that incorporates relative distance information, i.c.
the intervals [0.95D, 1.05D) and (0.95Dmmin 1.05D
fk.z: C'fks 24 -Pk)
3

---

Then radiance MLP U, decodes the RGB radiance at location 2; using an interpolation feature fz.- which is obtained from weighted interpolation according to inverse distance weights and neighboring features:
Wk
1
f#.
fae wi
(4)
Wk
1pp xall
U(2z))
(5)
If no neighbors are found, the occupancy a at I; is set to zero. Otherwise, we regress a using an occupancy MLP G at i₁. We follow a similar inverse iistance-based weighting interpolation as radiance, instead of interpolating at feature level. We decode for each neighbor and finally interpolate them:
Tk G(fr...)) Wk
(6)
ok:
(7)
Wk
Next we use Cz Oz, regressed from MLPs to estimate per-point weight ag,, Og, is regarded as the opacity at Ti, or the probability of a ray terminates at this point and 2r, is the depth of point zy- Depth map and color map can be rendered by calculating depth and radiance expectations along rays
as in Eq. 8
Niotal
Niota!
D=
Σ
i
Ozz, Cz.1
(8)
Mapping and Tracking. We use rendering difference loss as deseribed in Eq. that consists of geometric loss and photometric loss during the mapping process. For a new-coming frame, we sample M1 pixels optimize point-anchord first frame, we need to perform a good initialization at few higher cost of M3 pixels and around 3000~5000 optimization steps to ensure smooth following processing. For subsequent mapping, we select pixels uniformly from the current frame and 5 co-visible keyframes. We find that joint mapping can effectively stabilize the tracking process.
features fi and parameters of MLP C.U.G. With the
M Dm - -Dml + >1 - -iml.
Lmapping
Σ m==
M1
where Dm. Im represent ground truth depth and color map, Dm im are corresponding rendering results, and Å is the loss balance weight. During the tracking process, we backpropagate farther to optimize camera extrinsics {g.t} while keeping features and MLPs fixed, where is quaternion rotation and the translation vector. We sample M2 pixels across new-coming frame and assume the zero motion model where the initial pose of a new frame is identical to that of the last frame. Considering the strong non-convexity of the color map, only the geometry part included in tracking loss:
1 M2
Ltracking =
Dm -Dmb
(10)
M2
m=1
3.2 Loop Detection and Sub-Map Alignment
To align sub-maps and reduce accumulated pose drift, we perform loop detection and calculate relative poses between different sub-maps. For each keyframe in the sub-map, we associate it with a descriptor generated by the pre-trained NetVLAD ① model. We use cosine similarity between descriptors as the judgment criteria for loop detection. When the camera moves too much between frames, pose optimization tends to fall into local optimum, so we use two similarity thresholds Xfiner Acoaree to find matching pairs that have enough overlap and ensure correct loop relative pose. Given two sub-maps, Mi and M2, we find out loop frames with largest overlap. {1 _ with similarity greater than Xyines and small overlap loop frames, {(4T) 1;²),.., ..., (1
) with similarity greater than Acvarse but less than Afine For pairs { _d2), we take pose of p as initial pose and 122 as the reference frame. The tracking method described in 3.1 can be used to obtain the relative pose measurement between MMMMM2H We can use this relative pose to perform 3D

---

Free fusion
Feature grid misaligned
Agent
(4)
(1)
Agent
Figure 2: Voxel Grid Limitation. (a) If the initial coordinate system is uniform, the neural field representations of existing neural SLAM such as voxel grid will suffer from misalignment during sub-map fusion.
(b) The point cloud is not restricted by 3D geometry shape and can be fused freely. After sub-map fusion, We can query neighbors at i; from observations of Agentl (green point cloud) and Agent2 (purple point cloud).
rigid transformation on sub-maps, so that all sub-maps are set in the same global coordinate system, achieving the purpose of sub-map alignment shown in Fig. 2
Ltoop (Tr)
1
Σ
D (T,) Dmll
(11)
M2
where T, is the relative pose between M1 and .M2. and D, δ the rendered depth and ground truth depth. For loop frames with small overlap ( 7 only use them perform pose graph optimization (refer to 3.4). Sub-map fusion will lead to neural point redundancy in specific areas, which imposes burden computing and memory. Due the sparsity of the neural point cloud, we adopt grid-based filtering strategy, that is, we perform non-maximum suppression based on the distance from neural point the center of p cube.
3.3 Distributeddoo-CCettraaiiee Learning
To enhance consistency and coopera- tion, collaborative SLAM, we adopt a two-stage MLP training strategy. At the first stage (Distributed Stage), each image sequence is considered as a dis- crete individual with a unique group of MIPs (C,, UG, Gj} for sequential tracking and mapping. After loop de- tection and sub-map fusion, expect to share common MLPs across all se- quences (Centralized Stage). To this end, introduce the Federated learning mechanism which trains a single network in a cooperating shared way. At the same time as sub-map fusion, we average each group of MLPs and fine-tune the averaged MLPs on all keyframes to unify discrete domains. Subsequently, we iteratively transfer sharing MIPs to each agent for local training and average the local weights the final optimization result of sharing MIPs, shown in Fig. 3
Centralized
Sharing MLLPs
MLP-1
MUP-2
Figure 3: Two-stage Learning Strategy.
3.4 Pose Graph Optimization and Global Map Refinement
After front-end processing of all se- quences, including tracking, mapping, loop detection, and sub-map fusion, we establish global pose graph model, in which per-frame poses are nodes, se- quential relative pose, and loop relative pose are edges. The pose graph model carry out
Binding
illustrated Fig. global pose graph optimization across the entire pose graph, referring to tradi- tional visual SLAM, to force the estimated trajectory closer to the ground truth. Global pose graph
Single agent Pose Graph
Two-agent Pose Groph
Keyframe centric Model
Figure Pose Graph and Keyframe-centric Model.
5

---

optimization effectively mitigates cumulative error and improves tracking accuracy. We use the Levenberg M Marquarelt algorithm solve this nonlinear global pose graph optimization problem described by Eq. [2 where v is the set of nodes, F. is the set of sequential edges, Fi is the set of loop edges and A; represents the uncertainty of corresponding edges.
Σ v" = arg min 2 &.ER,
A,A7'es
(12)
Naturally, it is expected that the neural point cloud layout should be rearranged following global pose graph optimization. However, a world-centrie point cloud map obviously cannot allow such adjustment. To tackle this limitation, we propose keyframe-centric neural point field, where each 3D point is associated with a keyframe (Fig. 4). Benefitting from this design, we can refine our point cloud 3D locations according to optimized poses. Following global map refinement, we employ grid-based filtering to deal with neural point cloud redundancy occurring in local regions. Considering the slight mismatch between refined neural fields and decoders after global map refinement, we end up performing low-cost fine-tuning of the global neural point fields with fewer optimization iterations.
4 Experiments
Our CP-SLAM system supports both single-agent and multi-agent modes. Thus, we evaluate our proposed collaborative SLAM system in two aspects, both single-agent experiments with loop closure and two-agent experiments, of varying sizes and complexity. In terms of single agent, we generate datasets based on the Replica [35] scenes and then compare our method against recent neural and traditional RGB-D SLAM methods. For the two-agent side, since no collaborative neural SLAM work has emerged so far, we compare our method with traditional methods. We also conduct ablation studies to show the importance of modules the proposed system.
NVIDIA RTX3090 GPU. In the two-agent experiment, we need an additional RTX3090 as the central server. To encode higher frequency detail information, during inference, we impose position encoding on the origin neighbor feature fi and relative distance ||p; -x with an order of and 7. We utilize the FRNN library to query K 8 nearest neighbors
Implementation Details. CP-SLAM system runs RGB-D sequence
GPU. In all our experiments, we set Nacar 16, Nuni 4, A1 = 0.2, D, = 0.001m, r = 0.15m, P = 0. 0.14m, M₁ = 3000, M3 = 3136, M2 = 1500. We extract keyframe every 50 frames and perform map optimization and point cloud supplementation every 10 frames. For single-agent experiments, we optimize the neural field for 200 iterations. For two-agent experiments, considering that features anchored on neural points do not need to be trained from scratch after sub-map fusion, we reduce the number of iteration steps to 150. Further implementation details can be found in our supplementary material.
Baselines. In the single-agent experiment, because we use the rendered loop-closure data, we primarily choose the state-of-the-ard neural SLAM systems such as NICE-SLAM [50], Vox-Fusion 44 and
[4] for comparison on the loop-closure dataset. For the two-agent experiment, we compare our method with traditional approaches, such as CCM-SLAM 33 Swarm-SLAN and ORB-SLAM3 4
[7
Datasets. For reconstruction assessment, we utilize the synthetic dataset Replica 335 equipped with high-quality RGB-D rendering SDK. We generate 8 collections of RGB-D sequences, 4 of which represent single-agent trajectories, each containing I 550 RGB-D frames. The remaining 4 collections are designed for collaborative SLAM experiments. Fach collection is divided into 2 portions, each holding 2500 frames, with the exception of Office-O-C which includes 1950 frames per part.
Metrics. As a dense neural implicit SLAM system, we quantitatively and qualitatively mcasure its mapping and tracking capabilities. For mapping, we evaluate LI loss between 196 uniformly-samplee depth maps, which terms of 3D triangle mesh, we compute mesh reconstruction accuracy. For tracking, we use ATE RMSE, Mean and Median to comprehensively measure trajectory accuracy to prevent the negative impact caused by a few extreme outliers.
rendered from neural point field, and ground truth ones. Furthermore, in
4.1 Two-agent Collaboration
We provide the quantitative results of two-agent experiments on four scenes including Replica (35 Apartment-O, Apartment-1, Apartment-2 (multi-room), and Office-O--C (single-room). We take RGB- based CCM-SLAM [3], RGBD-baasdd Swarm-SI.AM [7 and traditional ORB-SLAM3 4 for comparison. Table. reports the localization accuracy of different methods. Despite being affected

---

Office-O-C
Method
Part
Apartment-1
Apartment-2
Apartment-0
(Single Room)
RMSE[cm]] Mean
[am]] Median [em]]
CCM-SLAM ORB-SLAMS
2,12/1.94/1.7 .4/1,74
0.51/0.45/0.4
-/---
184/8.23/6.41
4
4.93//.65/5.00
1.35/1.05/0.cr 05/0..65
0.67/0.58/0.47
0.66/0.62/0..
Swarm-SLAs SLLM
17
Part
4.62/4.17/33
2.69/2.48/2.34
1.61/1.3/11.0
1.0700.660.90
Ours (w/o)
..15/0.990.88 88
1.45/1.34/1.1
0.70/0.48/0.27
0.71/0.62/0.6%
Ours (w/
1.11/0.95/0881
1.41/1.30/1.36
0.62/0.47/0.30
0.50/0.46/0.55
CCM-SLAM
9.31/6.36/555
0.48/0.43/0.38
-H--
0.76/0.36/0.16
ORB-SLAMB
093//400/3.88
1.36/1.2/1.11
1.46/1.11/0799
0.54/0.49/0.44
Swarm-SLAM
Part 2
6.50/5.27/4.39
8.53/7.59/7.16
1.98/14480.94
1.76/1 .55/1183
Ours (w/o)
2.12/2.05/2.23
2.54/2.45/2.60
1.61/1.55/1. .70
1.02/1 .03/0.00000 99
Ours (w/)
1.72/166//146
2.41/2.33/2.444
1.28/1.17/1.37
1.79/0.74/0.70
CCM-SLAM
3.71/4.15/3.6
0.49/0.44/0.3
-/---
5.30/4.291
ORB-SLAMS
4.93/4.35/4441
1.36/1.15/0.88
1. 1.07/0.85/0.63
0.60/0.56/0.55
Swarm-SLAM
Average
5.56/4.72/4.15
5.61/5 04/4, 72
1.80/144// .22
1.42/1 .6/11 26/1.41
Ours (w/o)
1.64/L.5/11.6
2.00/1.900 .90/1.98
L.16/.02/0.99
0.86/0.8 1/0.83
Ours (w/)
1.42/1.28/1.14
1.91/1.82/1.99
0.95/0.82/0.84
0.65/0.60/0.63
Table 1: Two-agent Tracking Performance. ATE RMSE()), Mean(L) and Median(.) are used as evaluation metrics. We quantitatively evaluated respective trajectories (part and part 2) and average results of the two agents. Comparison between ours(w/o) and ours(w/) reveals the importance of global pose graph optimization for collaborative tracking. "_ indicates invalid results due to the failure of CCM-SLAM.
GT
Ours CCM-SLAM
Apartment-1
Apartment-0
(CCM-SLAM Failed)
Apartment-2
Office-O-C
Figure 5: Two-agent Trajectories on Scenes. CCM-SLAM relics too much on 2D geometric information that it has a large drift. In contrast, neural implicit system has robust performance.
by complex environments in multi-room sequences, the proposed system generally maintains better performance than other methods. It is worth noting that system. It lacks the capability process multiple sequences the same time, thus unable to overcome the efficiency bottleneck in scene exploration. Specifically, concatenate multiple image sequences and feed them into BRB-SLAM3. leveraging its "atlas" strategy for multi-sequence processing, Also, it can be seen that CCM-SLAM failed in some scenes because traditional RGB-based methods are prone to feature mismatching especially textureless environments. Fig, depicts the trajectories of each agent. Once two sub-graphs fused, only low-cost fine-tuning required to adjust two neural fields and corresponding MLPs into a shared domain. Afterward, these two agents can reuse each other's 's previous observations and continue accurate tracking. Shared MLPs, neural fields, and following global pose graph optimization make CP-SLAM system tightly collaborative system.
is not collaborative SLAM
4.2 Single Agent with Loop
In Table. 2 we illustrate the localization performance of our system operating in single-agent mode on loop closure datasets. Our method exhibits a notable superiority over recent methods, including NeRF-based and traditional ones, primarily attributed to the integration of concurrent front-end and back-end processing, as well as incorporation of neural point representation. Qualitatively, we present trajectories of Room-O-loop and Office-3-loop from NeRF-based methods in Fig. 6 The
7

---

Office--Hoop
Ours
NICE-SLAM
Vox-Fusion
Room- 1-ooop
Figure 6: Single-agent Trajectories on 4 Scenes. In comparison with frequent jitters in the
trajectories of the other two methods, our trajectory much smoother.
Method
Metric
Room-O-Joop
Room- I-loop
Office-O-loop
Office-3-loop
Average
RMSE [em]
1.27
1.74
2.27
3.19
2.12
NICE-SLAM
Mcan [em]
1.15
1.61
1.91
2.77
1.86
Median [cm]
1.09
1.66
1.82
2.28
1.71
RMSE [em]]
0.82
1.35
0.99
0.82
0.99
Vox-Fusion
Mcan [cm]
0.77
1.30
0.94
0.74
0.94
Median [cm]
0.78
1.25
0.95
0.73
0.93
RMSE [cm]
0.54
0.21
0.58
0.89
0.56
ORB-SLAME
Mcan [em]
0.52
0.19
0.51
0,80
0.51
Median [cm]
0.53
0.19
0.52
0.52
RMSF [cm]]
0.61
0.51
0.67
0.38
0.54
Ours(w/o)
Mcan [em]
0.56
0.48
0.32
0.50
Median [em]
0.54
0.52
0.67
0.27
0.50
RMSE [em]
0.48
0.44
0.56
0.37
0.46
(w/)
Mcan [em]
0.44
0.40
0.53
0.42
Median [cm]
0.43
0.46
0.56
0.27
0.43
Table 2: Single-agent Tracking Performance. Our system consistently yields better results compared with existing single-agent neural and traditional SLAM. In the fourth and fifth rows, we compare the accuracy of our system without and with the pose graph optimization module. Results are obtained in an origin-aligned manner with the EVO toolbox.
experimental results demonstrate that the concentration level of density energy around sampling points is critical for neural SLAM. NICE-SLAM (50) uses dense voxel grids, which contain large number of empty spaces, and sampling points in these empty spaces have almost no contribution to the gradient propagation. Vox-Fusion M4 has incorporated an important modification, implementing sparse grid that is tailored to the specific scene instead of a dense grid. However, grid nodes can only roughly placed near objects, which is unfavorable for tracking. In our point-based method, we ensure that neural field fits the real scene well. Feature embeddings can accurately include scene information. Concentratively distributed sample points and neural point based representation provide more exact gradients for pose backpropagation, which enables to surpass NICE-SLAM and Vox-Fusion lower resolution and memory usage. Moreover, we extend experiments on the TUM-RGBD real-world dataset, comparing with Co-SLAM [39] and ESLAM [12]. The results in Golle.3ill illustrate that our method has also achieved sfate-of-the-ar performance in the real-world setting, and the loop detection and pose graph optimization are equally effective for the real-world scene.
Table.
4.3 Map Reconstruction
The results in Table. present quantitative analysis of the geometric reconstruction produced by our proposed system in comparison to NICE-SLAM (50) and Vox-Fusion [44 In our approach, we render depth maps and color maps every frames throughout the entire predicted trajectories and utilize TSDF-Fusion (built-in function in Open3D [49] library) construct mesh map. In

---

Method
fri-desk (w/o loop)
fr2-xyz (w/o loop)
fr3-office (w/ loop)
Average
Co-SLAM 39
7.10/6.83/6.79
4.05/3.76/3.47
5.58/5.06/4.57
5.58/5.22/4.95
ESLAM 12
6.81/6.56/6.88
Fail/Fail/Fail
4.23/3.91/3.73
Ours
7.84/7.34/7.12
3.93/3.50/3.29
3.84/3.47/3.39
5.20/4.77/4.60
Table 3: Real-world Tracking Performance. In this real-world experiment, we can find that CP-SLAM still performs the best. Besides, ESLAM fails in the fr2-xyz because of OOM (out of memory). indicates that metrics cannot be evaluated due to ESLAM failures.
Method
Metric
Room-O- Oolo
Room- | 1loop
Office-0-
Office 3-loop
Average
Depth LI [om]] +
1.54
1.00
0.93
2.06
1.38
NICE-SLAM 50
Acc. [em]
3.30
3.19
2.88
3.95
3.33
Depth L1 [em]
0.77
1.30
0.94
0.74
0.94
Mox-Fusion 44
Acc. [ew]] 4
2.25
1.67
1.68
2.31
1.98
Depth LI [om]]
0.32
0.23
0.22
0.76
0,38
Ours
Acc. [em]] +
1.53
1.20
1.21
1.7
1.41
Table 4: Reconstruction Results. Our system has a more powerful geometric reconstruction capability than existing methods.
Vax-Fusion
Ours
Gdl
Figure 7: Reconstruction. Collaborative reconstruction results of Apartment-1 (a) and Apartment-O (b) show good consistency. (c) It can seen that our system also achieves more detailed geometry reconstruction in four single-agent datasets, c.g., note folds of the quilt and the kettle in red boxcs. Holes in the mesh reconstruction indicates unseen area in our datasets.
NICE-SIAM,, several metrics are employed for mapping evaluation, namely depth L.1 loss, mesh accuracy, completion, and completion ratio. In our loop datasets, scenes are not completely scanned, which leads to holes in mesh reconstruction. Therefore, our comparative experiments mainly focus on depth LI loss and mesh accuracy. Fig. quuaitaaiv compares the single-agent reconstruction results of three methods and shows our collaborative mapping results. Evidently, our method achieves more detailed geometry across all datasets.
4.4 Ablation Study
In this section, we examine some modules and designs in our system to prove their importance and the rationality of our pipeline.
Pose Graph Optimization. In this section, we conducted ablation experiments on the PGO mod- report results of ours(w/o PGO) and ours(w/ PGO) in single-agent and
ule. Table. and Table. 2 two-agent cases respectively. With the help of PGO module, the average positioning accuracy in the single-agent experiment decreases by 10%, while that in the two-agent experiment decreases by 13%. Map Refinement. As shown in Fig. ([a), we qualitatively illustrate the neural point field layout before and after map refinement in MeshLab. We can observe that refined neural point cloud fits the ground truth mesh better.
Sampling Concentration. The density concentration near sampling points a keypoint that deter- mines the performance of neural SLAM. We design set of experiments with different sampling interval lengths on Replica Room-O-Loop sequence. As shown Fig. B(b), tracking accuracy and depth L11 loss consistently drop to 1.0cm and 0.6em as sampling points gradually diverge. This experiment fully verifies the theory in Section. 4.2
Neural Point Density. In CP-SLAM, employed a fixed-size cubic cell within the filtering strategy

---

Metric
/= 10cm
p=14cm
p=18cm
p=22cm
RMSE [evr]] +
0.83
0.65
0.86
1.12
Ours (w/)
Mean [em]] J
0.77
0.58
0.76
1.01
Median [em] 4
0.83
0.51
0.71
0.94
Table 5: Neural Point Density Analysis. Results indicate that the point cloud density should be an
appropriate level, neither too high nor too low. Empirically, we have found that our system achieves the best performance when the cubic cell is set p 14em.
Method
Tracking/Frame
Mapping/Frame
MLP Size
Feature Size
MICE-SLAM 50
1.77s
11.26s
0.58 0.58510'5
238.88MB
Fex-Fusion 44
0.36s
0.83s
2.73x 10"5
0.15MB
Ours
0.30s
10.10s
1.36x 10"5
0.62MB
Table 6: Runtime and Memory Analysis.
Refine
04
Mean Ertr
02-
Depth LJ Eoor
0.1D
0.4D
0.6D
0&5
(a)
(b)
Figure 8: Map Refinement and Sampling Concentration Ablation. (a) From a bottom view of Apartment-1,
observe that there is offset in the unrefined neural point cloud( while the refined neural point cloud(pink) fits well. (b) As the sampling points diverge, the error rises.
to adjust the neural points, i.e., only the neural point closest to the center of a cubic cell is retained. To further explore the impact of point cloud density on tracking accuracy. we compared performance in the original Replica RoomO scene across various sizes of the cubic grid. The ablation study results in Table. de that when the number of neural points small, they are not enough encode detailed scene geometry and color due to inadequate scene representation. Conversely, an excessive number of neural points obviously extends the learning time to converge. We empirically found that the setting 14em worked consistently well on all the test scenes with different complexities in our experiment.
Memory and Runtime Analysis. We evaluated the runtime and memory consumption of our system on Replica Office-O-loop scene compared to NICE-SLAM and Vox-Fusion. We report single-frame tracking and mapping time, the size of MLPs and memory footprint of the whole neural field in The huge feature size in NICE-SIO -LAM due to its dense hierarchical feature grid.
Table. 6
5
Conclusion
We have proposed CP-SLAM, the first dense collaborative neural SLAM framework based on a novel neural point based representation, which maintains complete front-end and back-end modules like traditional SLAM systems. The comprehensive pipeline enables our system to outperform the state-of-the-art methods in both localization and reconstruction. One limitation of our method is its requirement for considerable GPU resources to operate multiple image sequences. Also, our system has slightly weaker hole-filling ability in unobserved regions than feature grid-based methods, which arises from the fact that neural points are distributed around the surfaces of observed objects, encoding surrounding scene information within fxxedradius sphere. Moreover, the relative pose computation in the loop closure relies on the existing rendering- based optimization, which may be inaccurate for large viewpoint changes thus leading to drifting of map fusion. Hence, it is interesting to design a lightweight system with a coarse-to-fie pose estimation for future work.
Acknowledgment:: This work was partially supported by the NSFC (No. 62102356).
10

---

References
[1] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5297-5307, 2016.
[2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1854,,864, 2021.
[3] Michael Bloesch, Sammy Omari, Marco Hutter, and Roland Siegwart. Robust visual inertial odometry using direct ekf-based approach. 2015 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 298 304. IEEE, 2015.
[4] Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan D Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics,
2021.
[5] Rohan Chabra, Jan E Lenssen, Eddy 1lg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pages 608 625. Springer, 2020.
[6] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5939-5948, 2019.
[7] Chi-Ming Chung, Yang-Che Tseng, Ya-Ching Hsu, Xiang-Qian Shi, Yun-Hung Hua, Jia-Fong Yeh, Wen-Chin Chen, Yi-Ting Chen, and Winston Hsu. Orbeez-slam: real-time monocular visual slam with orb features and nerf-realized mapping. arXiv preprint arXiv:2209.1322 2022.
[8] Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt II 13, pages 834-849. Springer, 2014.
[9] Michael Grupp. evo: Python package for the evaluation of odometry and slam. https 2017.
//github.
[10] Xiyue Guo, Junjie Hu, Hujun Bao, and Guofeng Zhang. Descriptor distillation for efficient multi-robot slam. arXiv preprint arXiv:2203..0 08420, 2023.
[11] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6001-6010, 2020.
[12] M. M. Johari, C. Carta, and F. Fleuret. ESLAM:
Efficient dense slam system based hybrid representation of signed distance fields. In Proceedings of the IEEE international conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[13] Mohammad Mahdi Johari, Camilla Carta, and François Fleuret. Eslam: Efficient dense slam sys- tem based on hybrid representation of signed distance fields, arXiv preprint arXiv:2211.11704,
2022.
[14] Marco Karrer, Patrik Schmuck, and Margarita Chli. slam. IEEE Robotics and Automation Letters, 3(4):7722-7769, 2018.
visual-inertial
[15] Georg Klein and David Murray. Parallel tracking and mapping on camera phone. In 2009 8th IEEE International Symposium on Mixed and Augmented Reality, pages 83-86. IEEE, 2009.
[16] Xin Kong, Shikun Liu, Marwan Taher, and Andrew J Davison. vmap: Vectorised object mapping for neural field slam. arXiv preprint arXiv:2302.0188. 2023.

---

[17] Pierre- Yves Lajoie and Giovanni Beltrame. Swarm SLAMM Sparse Decentralized Collaborative Simultaneous Localization and Mapping Framework for Multi-Robot Systems. 2023.
[18] Pierre- Yves Lajoie, Benjamin Ramtoula, Yun Chang, Luca Carlone, and Giovanni Beltrame. Door-slam: Distributed, online, and outlier resilient slam for robotic teams. IEEE Roboties and Automation Letters, 5(2): 165611666166,, 2020.
[19] Stefan Leutenegger, Simon Lynen, Michael Bosse Roland Siegwart, and Paul Furgale. Keyframe-basee based Journal of Robotics Research, 34(3):31 (113):31--333 4-44, 2015.
odometry using nonlinear optimization. The International
[20] Fu Li, Shaowu Yang. Xiaodong Yi, and Xuejun Yang. Corb-slam: collaborative visual slam system for multiple robots. In Collaborative Computing: Networking, Applications and Worksharing: 13th International Conference, Collaborate 2017, Edinburgh, UK, December 11-13, 2017, Proceedings 13, pages 480-490 Springer, 2018
[21] Hai Li, Xingrui Yang, Hongjia Zhai, Yuqian Liu, Hujun Bao, and Guofeng Zhang. Vox-surf: foxel-based implicit surface representation. IEEE Transactions on Visualization and Computer Graphics, 2022.
[22] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019-2028, 2020.
[23] Simon Lynen, Markus W Achtelik, Stephan Weiss, Margarita Chli, and Roland Siegwart. robust and modular multi-sensor fusion approach applied to mav navigation. In 2013 IEEE/RS/ international conference on intelligent robots and systems, pages 3923-3929, IEEE, 2013.
[24] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajaadi, Jonathan T Barron, Alexey Dosovit- skiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 7210-7219, 2021.
[25] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 16190-16199, 2022.
[26] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[27] Anastasios Mourikis and Stergios Roumeliotis. A multi-state constraint kalman filter for vision-aided inertial navigation. In Proceedings 2007 IEEE international conference robotics and automation, pages 3565 3565-3572. IEEE, 2007.
[28] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: versatile and accurate monocular slam system. IEEE transactions on robotics, 3175):1147-116,, 2015.
[29] Raul Mur-Artal and Juan D Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics, 3355) 2017.
[30] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Con- volutional occupancy networks. In Computer EsiionEECY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IIII 16, pages 523-540, Springer, 2020.
[31] Tong Qin, Peiliang Li, and Shaojie Shen. Vins-mono: robust and versatile monocular visual-inertial state estimator. IEEE Transactions Robotics, 1004),000 2018.
[32] Antoni Rosinol, John J Leonard, and Luca Carlone, Nerf-slam: Real-time dense monocular slam with neural radiance fields. arXiv preprint arXiv:2210.136411 2022.
12

---

[33] Patrik Schmuck and Margarita Chli. Cem-slam: Robust and efficient centralized collaborative monocular simultaneous localization and mapping for robotic teams. Journal of Field Robotics, 36(4))763-781, 2019.
[34] Patrik Schmuck, Thomas Ziegler, Marco Karrer, Jonathan Perraudin, and Margarita Chli. Covins: Visual-inertial slam for centralized collaboration, In 2021 IEFE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pages 171-176. IEEE, 2021.
[35] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: digital replica of indoor spaces. arXiv preprint arXiv: 0etiv:/9060000 /906605797 2019.
[36] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew Davison. imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 6229 6238, 2021.
[37] Zachary Teed and Jia Deng. DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. Advances in neural information processing systems, 2021.
[38] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5481-5490 IEEE, 2022.
[39] Hengyi Wang, Jingwen Wang, and Lourdes Agapito. Co-slam:
Joint coordinate and sparse parametric encodings for neural real-time slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13293-13302, 2023.
[40] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 10689, 2021.
[41] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 2021.
[42] Liu Xinyang, Li Yijin, Teng Yanbin, Bao Hujun, Zhang Guofeng, Zhang Yinda, and Cui Zhaopeng. Multi-modal neural radiance field for monocular dense slam with a light-weight tof sensor. In International Conference on Computer Vision (ICCV), 2023.
[43] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5438-5448, 2022.
[44] Xingrui Yang, Hai Li, Hongjia Zhai, Yuhang Ming, Yuqian Liu, and Guofeng Zhang. Vox- fusion: Dense tracking and mapping with voxel-based neural implicit representation. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 499-507. IEEE, 2022.
[45] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 24:4805-48155 2021.
[46] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33:2492-2502 2020.
[47] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung- Yi Lin. inerf: Inverting neural radiance fields for pose estimation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1323-1330. IEEE, 2021.
[48] Li Yijin, Liu Xinyang, Dong Wenqi, Zhou han, Bao Hujun, Zhang Guofeng, Zhang Yinda, and Cui Zhaopeng. Deltar: Depth estimation from a light tof sensor and rgb image. In European Conference on Computer Vision (ECCV), 2022.
13

---

[49] Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing, arXiv: 098400947, 2018.
[50] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11776-12796, 2022.
14


Reviews:
[{"review_id": "t28WCfx3Td", "reviewer": "Reviewer_Zpdr", "summary": "There have been some papers on getting dense Visual SLAM with neural fields published in the last couple of years. In this work, the authors proposed a set of additions to prior pipelines that get the NN SLAM closer to the traditional approaches concerning the overall framework. Specifically, the authors addressed loop detection, pose graph optimization, bundle adjustment, and collaborative SLAM.", "questions": "1) I would like the authors to explain what is the motivation for the inclusion of NetVLAD. Can't those descriptors be obtained from the neural field?\n2) It is written that a pre-trained NetVLAD was used. It would be important to give more details about this training and how it generalizes to different scenes.\n3) Usually, with classical V-SLAM, the bundle adjustment estimates both the pose of the cameras and the 3D map simultaneously. It seems that the authors are doing this alternatively. I would like to see some comments on this.\n4) Why are the results for Apartment 2 so bad for the proposed approach? I would like the authors to comment on this.\n5) It seems that traditional V-SLAM methods using RGB-D are missing (such as OrbSLAM). I would like to see some comments on this.\n6) Maybe I missed something. But it is not clear to me how is the error in Tab. 2 computed.", "limitations": "Limitations are adequately addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Visual-SLAM with neural fields has the huge benefit of providing a dense map of the environment. The topic is relevant to computer vision, robotics, and I believe it fits the NeurIPS scope.\n2) The paper is well-written, easy to follow, and well-organized. Related work is OK.\n3) Although the technical novelties are relatively limited, in my opinion, the contributions to the pipeline are significant.", "weaknesses": "1) As written above, the technical contributions are limited. The paper focuses more on adding new steps to prior Visual SLAM pipelines.\n2) The inclusion of the NetVLAD (not new) as a tool for loop detection looks like a trick. It is not elegant, in the sense that it is a parallel feature extractor that (maybe) could be done using the neural field encoders.\n3) Only one dataset and 4 sequences are used, which is small concerning quantity and variety.\n4) I would prefer to see more comparisons with previous traditional methods (without neural fields).\n5) Figures 5 and 6 are not easily readable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and for appreciating our contributions and writing. Here are our responses to your comments.  \n**1. The paper focuses more on adding new steps to prior Visual SLAM pipelines. (W1)**   \nWe aim to build a complete collaborative NERF-based SLAM system, focusing more on system-level development. Such work is missing in the field of NERF-based SLAM. CP-SLAM is the first collaborative neural implicit SLAM system, consisting of neural point-based odometry, loop detection, sub-map fusion, and global pose graph optimization. In collaborative NERF-based SLAM, in addition to the lack of appropriate neural field representation, it is not feasible to simply stack and cascade traditional modules in a plug-and-play manner. Consequently, we have introduced technical enhancements to both the neural field representation and the pipeline of CP-SLAM, such as a novel keyframe-centric neural point field and a novel learning framework to maintain global consistency. \n\n**2. Why not use the neural field encoders, instead of NetVLAD, for loop detection? (W2+Q1)**   \n\nWe have simply tried to take neural point features as descriptors for loop detection. However, neural features only encode local surrounding information, rather than a global descriptor like NetVLAD. This leads to poor distinctiveness of these neural features and further results in false positive matches which makes the loop detection unstable.\n\n**3. Dataset is limited. (W3)**  \nWe have experimented on the TUM-RGBD real-world dataset, and the results are shown in **Tab.G3** in the global response. Following recently proposed NERF-based SLAM works, we chose three scenes: fr1-desk, fr2-xyz, and fr3-office (with loop closure) in TUM-RGBD. The results in **Tab.G3** illustrate that CP-SLAM has also achieved state-of-the-art performance in the real-world setting, and the loop detection and pose graph optimization are equally effective for the real-world scene. It is worth noting that we compared the most recently proposed SOTA methods Co-SLAM [Wang et al. 2023] and ESLAM [Johari et al. 2023] in the TUM-RGBD experiment, and ESLAM encountered an out-of-memory (OOM) issue in the fr2-xyz.\n\n**4. More comparisons with traditional methods should be added. (W4+Q5)**  \nWe have compared with two more baselines including ORB-SLAM3 [Campos et al. 2021]  and Swarm-SLAM [Lajoie et al. 2023] for the two-agent experiment to demonstrate our performance and the corresponding results are shown in **Tab.G1** in the global response. It is clear that our method still performs the best. To be noted, ORB-SLAM3 is not a collaborative SLAM system. It lacks the capability to process multiple sequences at the same time, thus unable to overcome the efficiency bottleneck in scene exploration. We evaluate ORB-SLAM3 just for a more comprehensive comparison.\n\n**5. Figures 5 and 6 are not easily readable. (W5)**  \nThank you for your comment, we have improved these two figures for a better visual experience, please refer to **Fig.1** and **Fig.2** in the attached **`pdf'** in our global response.\n\n**6. Training details and generalization of NetVLAD. (Q2)**  \nThe pre-trained NetVLAD model used in our paper comes from the open-source project proposed by Paul et al. In our test, we observed its robustness across various scenes and insensitivity to hyperparameters.  Actually, the pre-trained NetVLAD was also normally adopted for loop detection in other SLAM systems like DOOR-SLAM [Lajoie et al. 2020] and DSLAM [Cieslewski et al. 2018]. \n\n**7. Separate pose and map optimization. (Q3)**   \nDuring mapping, we select multiple frames for joint optimization based on the covisible correlations. Considering the limitation of GPU memory, the total number of rays is fixed, that is, only a few rays can be allocated to each frame. In this case, considering the irregular distribution of neural points compared to the regular feature grid, the pose adjustment is prone to the sub-optimal solution if it is done together with mapping, so for more stable pose optimization, we choose to perform tracking separately with more rays after mapping. \n\n**8. Pose drift in Apartment-2. (Q4)**  \nExisting NERF-based SLAM methods often construct an L1 loss between rendered depth and the ground truth depth, relying on local depth variance to guide pose estimation towards a correct solution. However, in the **`Apartment2-part2'** sequence, the camera frequently moves parallel to walls with uniform depth, aggravating non-convexity in the depth loss and resulting in ambiguous solutions. This is a common problem in recently proposed NERF-based SLAM approaches, and we hope to overcome this bottleneck in future work.\n\n**9. Error calculation method in Tab.2. (Q6)**  \nThe results in Tab.2 were obtained in an origin-aligned manner with the **EVO** toolbox for proper drift/loop closure evaluation. Specifically, after aligning the origin poses, we calculate the absolute trajectory error (ATE) for each pose and compute the RMSE/Mean/Median values. "}, {"review_id": "bq60ybwQCJ", "reviewer": "Reviewer_1EKn", "summary": "The paper presents CP-SLAM, a collaborative neural implicit SLAM system developed by the authors. CP-SLAM incorporates neural point-based odometry, loop detection, sub-map fusion, and global refinement. Additionally, the authors propose a novel neural point 3D scene representation with keyframes to enhance map fusion and adjustment. They introduce innovative learning and optimization frameworks to ensure accurate and consistent 3D mapping for cooperative localization and mapping. The performance of CP-SLAM is evaluated on various indoor RGB-D sequences, showcasing its superior mapping and camera tracking capabilities. The contributions made by the authors significantly advance the field of SLAM and offer state-of-the-art solutions.", "questions": "1. How does CP-SLAM handle scalability in terms of the size and complexity of the datasets?\n2. Can CP-SLAM be applied to real-world datasets to evaluate its performance in practical scenarios?\n3. Are there any specific limitations or challenges in the implementation of CP-SLAM that need to be addressed?\n4. Can Nicer-SLAM and DROID-SLAM be compared as a contrast?\n5. The article mentioned \"Naturally, it is expected that the neural point cloud layout should be rearranged following global pose graph optimization. However, a world-centric point cloud map obviously cannot allow such adjustment. To tackle this limitation, we propose a keyframe-centric neural point field, where each 3D point is associated with a keyframe\". Can the authors better explain why a world-centric point cloud map obviously cannot allow such adjustment? \n\nSincerely,\n", "limitations": "The limitations have been discussed in the conclusion section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. CP-SLAM introduces a novel neural point-based representation, enabling dense collaborative neural SLAM.\n2. CP-SLAM retains the front-end and back-end modules like traditional SLAM systems, providing a comprehensive pipeline.\n3. CP-SLAM outperforms state-of-the-art methods in both localization and reconstruction. The CP-SLAM system supports both single-agent and multi-agent modes, allowing for versatile evaluation.\n4. The system is evaluated using datasets based on the Replica scenes, providing a realistic and representative testing environment.\nCP-SLAM is compared against recent neural implicit RGB-D SLAM methods in single-agent experiments, showcasing its performance in loop closure.\n5. In the two-agent experiments, CP-SLAM is compared with a traditional method, highlighting its collaborative capabilities.\n6. The ablation studies are conducted to demonstrate the importance of different modules in the proposed system.\n7. The implementation details, such as GPU specifications and parameter settings, are provided, ensuring reproducibility.", "weaknesses": "1. CP-SLAM requires significant GPU resources to handle multiple image sequences. The relative pose computation in loop closure relies on existing rendering-based optimization, which may be inaccurate for large viewpoint changes and lead to map fusion drifting.\n2. The comparison with traditional methods in the two-agent experiments may not fully represent the performance of other collaborative neural SLAM approaches.\n3. CP-SLAM's performance is limited by its GPU resource demand, making it less suitable for resource-constrained devices or environments.\n4. Currently, a large-scale experiment is not conducted, but for multi-agents, this is an important point to consider.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading and constructive comments on our paper, and noting the novelty of our CP-SLAM system and its superiority. Here are our responses to your comments.\n\n**1. Large viewpoint change is challenging for NERF-based SLAM methods. (W1)**\n\nAs pointed out in Section.5 in our paper, the rendering-based optimization is limited in the face of large viewpoint changes, which remains a bottleneck for other existing works, such as NICE-SLAM and Vox-Fusion. We conduct an ablation study on viewpoint difference in **Fig.4** in the attached **'pdf'** in the global response. It may be partially addressed by the particle filtering algorithm proposed in the latest Loc-NeRF [Maggio et al. ICRA 2023].\n\n**2. More comparisons with traditional methods should be added in the two-agent experiments. (W2)**\n\nDue to the word limit, please refer to **'Comparison with additional traditional methods'** and **Tab.G1** in our global response.\n\n**3. GPU resources are a limitation for CP-SLAM. (W3)**\n\nIt is indeed a common weakness in existing neural implicit SLAM works and our CP-SLAM, and we expect further improvements to be made in the future.\n\n**4. A large-scale experiment is not conducted in this paper. (W4)**\n\nIn our research, we indeed struggled with limited data resources, and we have not been able to find a public large-scale dataset suitable for collaborative NERF-based SLAM. Multi-agent datasets were mainly captured at large-scale outdoor scenes, such as urban streets. Affected by noisy depth, these available datasets are normally RGB-based, which is not compatible with our RGBD-based CP-SLAM. Moreover, when applied to outdoor scenarios, almost all existing NERF-based SLAM systems encounter pressing issues such as dynamic interference, unbounded scene, and fast motion.  These challenges deserve immediate and critical attention.  It is interesting to work on these limitations and generalize CP-SLAM to outdoor environments, which are out of our current scope.\n\n**5. How does CP-SLAM handle various scene sizes and complexity? (Q1)**\n\nIn terms of scene size, in principle, CP-SLAM only needs to incrementally add local neural points to support follow-up tracking. Facing large-scale scenes, our CP-SLAM is more scalable, particularly when compared to the feature grid-based method such as NICE-SLAM. In terms of complexity including extreme lighting and weak texture, NERF-based SLAM exhibits even better robustness over traditional methods, which is evidenced by the failure of traditional methods in Apartment-0 in Tab.1 of our paper.\n\n**6. Can CP-SLAM be generalized to real-world scenes? (Q2)** \n\nDue to the word limit, please refer to **‘Real-world experiment’** and **Tab.G3** in our global response.\n\n**7. Are there any specific limitations or challenges in the implementation of CP-SLAM that need to be addressed? (Q3)**\n\nIn collaborative NERF-based SLAM, in addition to the lack of appropriate neural field representation, it is not feasible to simply stack and cascade traditional modules in a plug-and-play manner. Consequently, we have innovatively proposed a keyframe-centric neural point field to address the limitation of sub-map fusion and a novel learning framework to maintain global consistency, etc.\n\n**8. Can Nicer-SLAM and DROID-SLAM be compared as a contrast? (Q4)**\n\nNicer-SLAM is a non-public and monocular RGB-based neural SLAM system, while our CP-SLAM is a collaborative RGBD-based system. Therefore, we only added Droid-SLAM as a new baseline, and we report experiment results from the official DROID-SLAM in the following table:\n\n|   Method    |    Room-0-loop     |    Room-1-loop     |   Office-0-loop    |   Office-3-loop    |      Average       |\n| :---------: | :----------------: | :----------------: | :----------------: | :----------------: | :----------------: |\n| DROID-SLAM  | **0.28/0.26/0.23** | **0.26/0.22/0.19** | **0.45/0.41/0.37** | **0.33/0.29**/0.30 | **0.33/0.30/0.27** |\n| Ours(w$/$o) |   0.61/0.56/0.54   |   0.51/0.48/0.52   |   0.67/0.63/0.67   |   0.38/0.32/0.27   |   0.54/0.50/0.50   |\n| Ours(w$/$)  |   0.48/0.44/0.43   |   0.44/0.40/0.46   |   0.56/0.53/0.56   | 0.37/0.31/**0.27** |   0.46/0.42/0.43   |\n\n**Table.A Single-agent Tracking Performance compared with DROID-SLAM**\n\n|   Method   |  Room-0-loop  |  Room-1-loop  | Office-0-loop | Office-3-loop |    Average    |\n| :--------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| DROID-SLAM |    58/6.47    |    3/4.17     |    34/8.12    |    61/7.99    |    46/6.69    |\n|    Ours    | **0.32/1.53** | **0.23/1.20** | **0.22/1.21** | **0.76/1.7**  | **0.38/1.41** |\n\n**Table.B Mapping Performance compared with DROID-SLAM**\n\nWe observe that CP-SLAM performs slightly worse than DROID-SLAM in terms of tracking, but much better than DROID-SLAM in reconstruction. In addition, DROID-SLAM cannot support multi-agent collaboration.\n\n**9. Why the world-centric model cannot allow neural point cloud adjustment after pose graph optimization? (Q5)**\n\nKeyframe-centric model is a crucial module in the proposed CP-SLAM system. Specifically, the neural point field is composed of two parts: the 3D point cloud and the corresponding feature embeddings which encode the surrounding regions. Following pose graph optimization (PGO), it is necessary to refine previously selected point positions based on more accurate poses, so as to reuse previously optimized feature embeddings. This requires a clear correspondence between 3D points and mapping frames. If a world-centric model is adopted, neural points cannot be refined along with camera poses due to the absence of such correspondences, so it may be possible to recompute all 3D points and learn all neural features from scratch for stable tracking and mapping after PGO."}, {"review_id": "LSKtqxbsax", "reviewer": "Reviewer_q5H1", "summary": "The paper introduces a neural implicit representation based collaborative SLAM technique. They introduce a learnable feature encoding for each point in the 3d scene based on neural rendering subsequently using it for depth and color map estimation. They introduce a centralized learning strategy to share MLPs and learn better weights. They also introduce a LM algorithm based global pose optimization and key frame centric point cloud adjustment.", "questions": "1. The paper is not clear whether it includes neural field generation time in tracking/mapping time while comparing with Nice SLAM. \n2. The paper lacks citations. For example, in the mapping and tracking section, a similar technique is used in Nice SLAM and I see no mention of it.", "limitations": "1.The paper does produce SOTA results on some of the scenes of the dataset chosen. No explanation is given to why it doesn't produce better results on the rest. \n2. The paper is not novel enough. It rather combines the best of many other works and produces good results.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The federated learning strategy improves the neural rendering performance. Map refinement based on keyframe centric neural point field.\nBy encoding scene geometry and appearance with 3D points, when camera poses are optimized, these features are easily adjusted as well.", "weaknesses": "1. The paper only evaluates on one synthetic datasets. It would be nice to see evaluations on more datasets \n2. Better and more robust neural rendering techniques could have been used to improve the radiance, depth and color rendering", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments! Here are our responses to your concern.\n\n**1. It would be nice to see evaluations on more datasets. (W1)**\n\nWe extended experiments on the TUM-RGBD real-world dataset, and the results are shown in the **Tab.G3** in global response. Following recently proposed NERF-based SLAM works, we chose three scenes: fr1-desk, fr2-xyz, and fr3-office (with loop closure) in TUM-RGBD. The results in **Tab.G3** illustrate that CP-SLAM has also achieved state-of-the-art performance in the real-world setting, and the loop detection and pose graph optimization are equally effective for the real-world scene. It is worth noting that we compared the most recently proposed SOTA methods Co-SLAM [Wang et al. 2023] and ESLAM [Johari et al. 2023] in the TUM-RGBD experiment, and ESLAM encountered an out-of-memory (OOM) issue in the fr2-xyz.\n\n**2. More advanced neural rendering techniques can be used to improve the rendering quality. (W2)**\n\nSeveral innovative and robust neural rendering techniques have been proposed subsequent to our submission. Nevertheless, given its nature as a collaborative NERF-based SLAM system, we focus more on the overall performance of tracking, mapping, and collaborative cooperation than rendering quality. However, it is interesting to explore more robust rendering techniques in the future.\n\n**3. It is not clear whether time evaluation includes neural field generation time. (Q1)**\n\nFollowing previous works, such as NICE-SLAM and Vox-Fusion, we evaluated the efficiency of mapping and tracking in Tab.4. All results in Tab.4 solely measure single-frame optimization time in tracking and mapping, excluding the neural field generation time (neural point back-projection and initial neural features inference from CNN).\n\n**4. Some citations are missing in the paper. (Q2)**\n\nIn Section 3.1, we described volume rendering, depth loss, and color loss used in tracking and mapping. We presumed these concepts as common sense reached in the NERF-based SLAM field, so we omitted relevant citations. Thanks for your comment, we have checked and added them in Section 3.1.\n\n**5. Novelty is limited. (L1)**\n\nWe aim to build a complete collaborative NERF-based SLAM system, focusing more on system-level development. Such work is missing in the field of NERF-based SLAM. we have introduced technical enhancements to both the neural field representation and the system-level pipeline, such as innovatively proposing a keyframe-centric neural point field and a novel learning framework to maintain global consistency. Although the technical novelties are relatively limited, we have made significant contributions to the pipeline as **Reviewer Zpdr** pointed out."}, {"review_id": "rmwnl8MJac", "reviewer": "Reviewer_zwcP", "summary": "The authors propose a novel collaborative RGB-D SLAM pipeline, where the map is represented by an implicit field based on Point-NERF that are transformed together with their keyframes allowing pose graph optimization. They use a loop closure detection based on learned features (NetVLAD) also for sub-map alignment. They propose a central federated learning based averaging and refinement of the distributed MLPs after submap alignment.", "questions": "Why is there another local finetuning and weight averaging done after the central averaging and finetuning?\nHow is the finetuning done after pose graph optimization? Is it the same as for mapping e.g. eq 9?\nWould it not be possible to run ORB-SLAM3 (https://arxiv.org/pdf/2007.11898.pdf) in a multi-map setting using depth from RGB-D as a baseline?\nWhy not evaluate on real data?\nHow is the TSDF fusion done to generate the final mesh?\nWhat is the average baseline between frames in the dataset?\nHow long are the trajectories?\n\nAfter rebuttal: The authors addressed all of my questions. Thank you.", "limitations": "One of the main limitations of framerate/view point change is addressed by the authors.. an initial experiment to showcase the effect would have been nice though.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The use of point based features nicely connects the generalization and compression of implicit mapping with the ability to do efficient map refinement. The results on tracking and reconstruction show a large improvement compared to existing works.", "weaknesses": "The explanations are sometimes a bit vague. E.g. line 101 what does it mean to incrementally project a pixel to a 3d location?\nThe initialization of system is not described well.\nWhile the authors describe the averaging and finetuning of the MLP weights, but do not discuss if and how the point features are shared between agents.\nEvaluation was only done on synthetic data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Here are our responses to your concerns.\n\n**1. What does it mean to incrementally project a pixel to a 3d location in line 101? (W1)**\n\nIn the SLAM task, both the map and trajectory are incrementally constructed over time. In Line 101, when a new mapping frame arrives, we will back-project a 2D pixel location $\\textbf{p}$ to a 3D point $\\textbf{P}$ based on the pose, intrinsics, and the depth map, as shown in the following formula:\n$$\n\\begin{equation}\n\\label{eq1}\n    P = D*K^{-1}*p\n\\end{equation}\n$$\n\nHere, $D, K$ represent the ground truth depth and intrinsics, and $p$ is the homogeneous pixel coordinate.\n\n**2. The initialization of the system is not described well. (W2)**\n\nThe initialization in NERF-based SLAM refers to utilizing the first frame to construct an initial neural field, similar to the mapping process, but with a few more optimization steps. Specifically, in CP-SLAM, we perform the initialization with 3000$\\sim$5000 steps. This practice can stabilize subsequent tracking and mapping with the help of a high-quality initial neural field.\n\n\n\n**3. Authors did not discuss if and how the point features are shared between agents. (W3)**\n\nIn our implementation, we built up a real-time updating feature list corresponding to 3D points one by one, and synchronize it to each agent during data exchange.\n\n\n\n**4. Can CP-SLAM be generalized to real-world scenes? (W4+Q4)**\n\nWe have experimented on the TUM-RGBD real-world dataset, and the results are shown in the **Tab.G3** in the global response. Following recently proposed NERF-based SLAM works, we chose three scenes: fr1-desk, fr2-xyz, and fr3-office (with loop closure) in TUM-RGBD. The results in **Tab.G3** illustrate that CP-SLAM has also achieved state-of-the-art performance in the real-world setting, and the loop detection and pose graph optimization are equally effective for the real-world scene. It is worth noting that we compared the most recently proposed SOTA methods Co-SLAM [Wang et al. 2023] and ESLAM [Johari et al. 2023] in the TUM-RGBD experiment, and ESLAM encountered an out-of-memory (OOM) issue in the fr2-xyz.\n\n\n\n**5. Why is the local finetuning and weight averaging done after the central averaging and finetuning? (Q1)** \n\nOnce loop closure is detected, we will conduct central averaging and finetuning to unify different models into a common domain. Then the finetuned model will be sent to each agent for continuous local tracking and mapping. Once the new keyframe arrives, we will conduct the local update/finetuning given the new observation and the finetuned local model will be sent to the center for central averaging as a new global model for subsequent tracking. This approach only requires lightweight data exchange which further improves real-time performance.\n\n\n\n**6. How is the finetuning done after pose graph optimization? Is it same as Eq. 9? (Q2)** \n\nFinetuning after pose graph optimization is taken as a larger-scale joint mapping. Specifically, we will select a certain number of keyframes from the keyframe pool, which contains keyframes from all agents, for a few-step mapping until the entire pool is traversed. It uses the same principle as **Eq.9**. \n\n\n\n**7. Would it not be possible to run ORB-SLAM3 as a baseline? (Q3)**\n\nAs far as we know, ORB-SLAM3 [Campos et al. 2021] is not a collaborative SLAM system. It lacks the capability to process multiple sequences at the same time, thus unable to overcome the efficiency bottleneck in scene exploration. As suggested, we tested ORB-SLAM3 with **‘Atlas'** mode, and we find that only when ORB-SLAM3 fails, it works by saving the disconnected map constructed prior to failure in the atlas as a candidate sub-map, and then creates a new sub-map for subsequent mapping and tracking. If a loop is detected between this new sub-map and one in the atlas, ORB-SLAM3 will perform sub-map fusion. \nFor a comprehensive comparison, we have sequentially connected multiple parts from a single multi-agent dataset. We reported multi-agent and single-agent tracking results in **Tab.G1** and **Tab.G2** in the global response. It is clear that CP-SLAM still achieves the best performance in most scenes.\n\n**8. How is the TSDF fusion done to generate the final mesh? (Q5)**\n\nIn our implementation, we invoked the built-in **`TSDF-Fusion'** function in Open3D library, supplying it with estimated poses and rendering depth to generate the final mesh.\n\n**9. Average baseslines and the length of trajectories in all datasets. (Q6+Q7)**\n\nAverage baselines and the length of trajectories in all datasets are shown in the table below:\n\n|                     | Room-0-loop | Room-1-loop | Office-0-loop | Office-3-loop |\n| :-----------------: | :---------: | :---------: | :-----------: | :-----------: |\n|     Length[$m$]     |    7.69     |    7.82     |     7.69      |     12.95     |\n| Baseline(mean)[$m$] |   0.0039    |    0.004    |    0.0039     |    0.0066     |\n\n**Table A.  Average baselines and the length of trajectories in single-agent datasets.**\n\n|                     |  Part  | Apartment-1 | Apartment-2 | Apartment-0 | Office-0-C |\n| :-----------------: | :----: | :---------: | :---------: | :---------: | :--------: |\n|     Length[$m$]     | Part 1 |    18.04    |    12.12    |    8.71     |    5.66    |\n|                     | Part 2 |    18.61    |    11.24    |    13.34    |    5.66    |\n| Baseline(mean)[$m$] | Part 1 |   0.0046    |   0.0048    |   0.0053    |   0.0037   |\n|                     | Part 2 |   0.0063    |   0.0040    |   0.0035    |   0.0037   |\n\n**Table B. Average baselines and the length of trajectories in two-agent datasets.**"}, {"review_id": "AFpDhMx45F", "reviewer": "Reviewer_46C6", "summary": "This paper proposed a collaborative RGB-D neural implicit SLAM method named CP-SLAM. CP-SLAM is based on neural point representation similar to PointNeRF, which is natually suitable for collaborative SLAM and loop closing. Based on this neural point representation, CP-SLAM is able to perform full SLAM with a single or multiple agents: tracking, mapping, loop detection, loop closing, submap merging, etc. Experiments show that CP-SLAM achieves SOTA results compared with previous collaborative SLAM and neural implicit SLAM methods.", "questions": "1. From Sec. 3.1 it seems that at each frame a new set of neural points are created densely via back-projection, and there doesn't seem to be any point merging scheme. Then how is CP-SLAM able to achieve 0.62MB parameters size in Tab. 4?\n\n2. Also from Sec. 3.1 it seems that the rendering loss will only optimize the neural features stored in the points but will never affect the locations of the points, which means once created the location of a point will never change unless a loop is detected and a Pose Graph Optimisation is performed. If this is the case, then the entire mapping process would be equivalent to simply back-projecting the depth images?\n\n3. How did you convert your point-based map to mesh for evaluation in Tab. 3?\n\n4. Following my question in 2: if CP-SLAM's map points are obtained explicitly from depth measurement, then it is supposed to have better accuracy than NICE-SLAM and Vox-Fusion (the same reason that TSDF-Fusion has better accuracy than neural SLAM methods). Thus, better accuracy numbers in Tab. 3 is not enough to prove CP-SLAM have \"more powerful geometric reconstruction capability\". Also it's not clear why the authors didn't report completion.\n\nAfter rebuttal: the authors have addressed all my questions and concerns.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Using neural point representation in the context of neural implicit SLAM is novel and the motivation of using it is also clear, i.e. more suitable for loop closing and sub-map merging.\n\n2. A full SLAM system that is capable of loop closing, Pose Graph Optimisation and sub-map merging. This is the first time I saw a full SLAM system in the context of neural implicit SLAM.\n\n3. The system seems well engineered, putting many components together and works effectively well.\n\n4. Competitive tracking and mapping results compared with previous collaborative and neural SLAM method.", "weaknesses": "The motivation is clear and the method is very well engineered, but from novelty perspective, all the components in CP-SLAM are already there with some incremental modification: e.g. the scene representation is from PointNeRF, Loop detection is from an off-the-shelf NetVLAD model, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and recognition of our strengths. Our responses to your concerns are as follows:\n\n**1. Some components in CP-SLAM are incrementally modified based on existing works. (W1)**\n\nWe aim to build a complete collaborative NERF-based SLAM system, focusing more on system-level development. Such work is missing in the field of NERF-based SLAM. We have introduced technical enhancements to both the neural field representation and the system-level pipeline, such as innovatively proposing a keyframe-centric neural point field and a novel learning framework to maintain global consistency. Although the technical novelties are relatively limited, we have made significant contributions to the pipeline as **Reviewer Zpdr** pointed out.\n\n**2. Is there a point cloud fusion strategy in CP-SLAM to reduce memory usage? (Q1)**\n\nIn Line 163, we briefly mentioned our point cloud filtering strategy, namely the sparsification strategy, and elaborated on it in **Supp.D**. Specifically, we adopt a grid-based filtering strategy, that is, we perform non-maximum suppression based on the distance from a neural point to the center of a $\\rho^3$ cube.\n\n**3. Is the entire mapping process in CP-SLAM equivalent to simply back-projecting the depth images? (Q2)**\n\nCP-SLAM is completely different from traditional visual SLAM methods. As a purely NERF-based SLAM system, 3D points serve solely as spatial carriers of neural feature embeddings for both photometric and geometric information. In CP-SLAM, the mapping process refers to the process of training neural features to better represent the appearance and geometry of the scene and improve the rendered RGB and depth images, rather than back-projecting the depth image. In other words, although the points' location does not change, the encoded scene geometry is changed and improved with the optimized neural features. We dynamically illustrate the above mapping process in our attached supplementary video. \n\n**4. How did you convert your point-based map to mesh for evaluation in Tab.3? (Q3)**\n\nIn our implementation, we invoked the built-in **'TSDF-Fusion'** function in Open3D library, supplying it with estimated poses and depth maps from neural volume rendering to generate the final mesh.\n\n**5. If 3D points in CP-SLAM are obtained from depth measurements,  the accuracy improvement in Tab. 3 is not enough. Why the completion is not reported in reconstruction? (Q4)**\n\nAlthough we utilize ground truth depth maps to obtain neural point locations, as mentioned above, 3D points serve solely as spatial carriers of neural feature embeddings in CP-SLAM. What we input into **`TSDF-Fusion'** is the rendered depth maps from the optimized neural field, not the measurements. In this setting, we achieved better mapping performance than NICE-SLAM and Vox-Fusion. In Lines 274 and 275, we mentioned the reason why we did not evaluate the completion metric. In our loop datasets, the scenes are not completely scanned along with our trajectories, which leads to holes in mesh reconstruction and makes this metric less objective. Therefore, we use 'Depth L1' and 'Acc.' as evaluation metrics of the reconstruction.\n"}, {"review_id": "re0koQmbZX", "reviewer": "Reviewer_jr2H", "summary": "The paper presents a collaborative implicit neural simultaneous localization and mapping (SLAM) system for RGB-D image sequences, which is able to merge sub-maps into a global map and also handle loop-closure for neural mapper for the first time. To facilitate the capability of doing loop-closure and sub-map merging, the authors introduce a neural point-based 3D scene representation, where each point maintains a learnable neural feature for scene encoding and is associated with a keyframe. The optimization follows the traditional SLAM’s graph optimization fashion. The experimental results on diverse datasets demonstrate the superior performance of the proposed method in both camera tracking and mapping.", "questions": "1.In the context of this paper, which relies on point features for establishing frame-to-frame correspondence, an important concern arises regarding the maintenance of continuous tracking. Unlike the traditional SLAM pipeline, where feature locations typically reside around corners, this paper selects points without necessarily adhering to this characteristic. Thus, it becomes crucial to address how this approach ensures the continuity of tracking throughout the SLAM process.\n2.Collaborative mapping can be done by recent work on No Pose Nerf:\n[]Bian, Wenjing, et al. \"Nope-nerf: Optimising neural radiance field with no pose prior.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\nThis paper should provide a comparison to show the advantage of the proposed method.\n3. This paper has too many contributions, which does not form a scientific paper but more an engineering paper. Not sure if this paper fall into the NeuralIPS group.", "limitations": "1. The writing should be improved, especially the equations should be better explained.\n2. More experiments and ablation studies should be provided to demonstrate the advantage of the proposed algorithms. \n3. The literature review needs to refer to more recent research.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel Neural Point-based 3D Scene Representation: A neural point cloud representation for scene encoding in the proposed framework is a significant contribution. This approach allows each point to carry encoded information, improving the system's ability to represent and cross-view association for large-scale tracking and pose refinement. \n2.Comprehensive Collaborative Framework: The authors provide a collaborative SLAM system with front-end and back-end modules, addressing essential aspects such as odometry, loop detection, sub-map fusion, and global refinement. This comprehensive approach ensures that the proposed system can handle various challenges typically encountered in SLAM tasks. \n3.Distributed-to-Centralized Learning Strategy: The incorporation of a distributed-to-centralized learning strategy is an effective solution for improving consistency and cooperation in collaborative implicit SLAM. \n4.Improved Performance: The experimental results on multiple datasets demonstrate the superiority of the proposed method in both camera tracking and mapping.", "weaknesses": "1.In the Introduction: This paper emphasizes the significance of targeting collaborative-SLAM, considering the substantial value already provided by loop-closure capability in effectively reducing pose drift. However, the evaluation of the paper becomes more challenging due to the necessity for additional comparative studies that can convincingly demonstrate the advantages offered by both approaches.\n\n2.In Section 2: For collaborative mapping, this paper should cite more recent works including:\n[1*] Campos, Carlos, et al. \"Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam.\" IEEE Transactions on Robotics 37.6 (2021): 1874-1890.\n[2*] Tian, Yulun, et al. \"Kimera-multi: Robust, distributed, dense metric-semantic slam for multi-robot systems.\" IEEE Transactions on Robotics 38.4 (2022).\n\nIt is also expected to provide comparative studies to the above researches.\n\n3.In Section 3: \n(1). Line 101: This paper introduced 4x4 patches to compute the point features. However, it misses details on there is a limitation on the number of patches. What if we increase the size of the patches? Will it impact the later performance? \n(2) This paper introduced so many symbols but without giving any explanation of the symbols, which makes it really hard to follow the paper. Please revise the whole paper and make them clear to the readers.\n(3) Equation 10: The explanation “Considering the strong non-convexity of the color map, only the geometry part is included in tracking loss” for not using color in the loss is not convincing. The authors are expected to provide ablation studies to prove the claim.\n\n4.In Section 4: More experiments are expected:\n(1). More comparative studies on Collaborative-SLAM are expected.\n(2). For single-agent cases: The local covisibility map can be formed as well; it would be interesting to only look at cases without a loop but having a local covisibility map.\n(3). Ablation study on “Distributed-to-Centralized Learning” is needed to prove why this is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and valuable suggestions. Here are our responses to your comments.\n\n**1. To demonstrate advantages from both collaboration and loop-closure capability, evaluation in this paper becomes more challenging. (W1)**\n\nIn CP-SLAM, loop detection and pose graph optimization (PGO) work together to mitigate pose drift, and collaboration mainly helps to improve exploration efficiency, but has little effect on pose drift. In Tab.2 and Tab.1, we perform comparative studies in  the **‘w/ and w/o PGO‘** setting and observed consistent improvement across all datasets. Overall these two experiments consistently demonstrate the effectiveness of our proposed pipeline.\n\n**2. This paper should cite and compare with more recent works, such as ORB-SLAM3 and Kimera-multi. (W2)**\n\nAs far as we know, ORB-SLAM3 is not a collaborative SLAM system. It lacks the capability to process multiple sequences at the same time, thus unable to overcome the efficiency bottleneck in scene exploration. As suggested, we tested ORB-SLAM3 for a comprehensive comparison. We have sequentially connected multiple parts from a single multi-agent dataset. We reported multi-agent and single-agent results in **Tab.G1** and **Tab.G2** in global response. In addition, as a visual-inertial method, Kimera-multi takes IMU data as input, thus we did not consider it as a comparable method.\n\n**3. In Section 3: (1). Line 101: How the performance is affected by the number of patches? (2) Some symbols are not clear. (3) Equation 10: Authors should perform an ablation study to confirm their point on non-convexity from color loss. (W3)**\n\n**(1)** In principle, the number of patches will not affect CP-SLAM in terms of memory usage and accuracy. Regarding memory usage, more patches bring more 3D neural points. However, we apply a filtering strategy to maintain point cloud sparsity, as indicated in **Supp.D.** From the perspective of accuracy, we empirically found that an increased quantity of neural points slightly decreases the tracking and mapping performance due to the increased complexity and redundancy of the neural point field. \n\n**(2)** Thank you for your comment, we will check and clarify all symbols to make our paper more readable. \n\n**(3)** We conducted an additional ablation study to confirm our viewpoints, as shown in **Fig.3** in the attached **'pdf'** in global response.\n\n**4. (1). More comparisons on collaborative SLAM are expected. (2). For single-agent cases: it would be interesting to take a local covisibility map into account. (3). Ablation study on “Distributed-to-Centralized Learning” is needed to prove its necessity. (W4)** \n\n**(1)** We added ORB-SLAM3 and Swarm-SLAM as new baselines for fully comparative studies. The results are shown in **Tab.G1** in global response, we can observe that our method still performs the best. \n\n**(2)** In CP-SLAM, we have used covisible correlations between keyframes. Specifically, we utilize multiple frames, including the current frame, the nearest keyframe, and three other covisible keyframes, for joint mapping, which ensures the consistency of the neural field. However, CP-SLAM which relies on neural features is completely different from the traditional visual SLAM that relies on the point cloud. In the volume rendering-based SLAM, the local covisibility map cannot replace the loop detection module. We present results for single-agent cases with joint mapping but without PGO in Tab.2. \n\n**(3)** In **Supp.C**, We discussed the necessity of `Distributed-to-Centralized' learning strategy. We have attempted to perform centralized learning from scratch, but its results were remarkably poor.\n\n**5. It is crucial to address how CP-SLAM ensures the continuity of tracking throughout the SLAM process. (Q1)**\n\nCP-SLAM is a collaborative NERF-based SLAM system, similar to NICE-SLAM, relying on volume rendering to optimize poses and the neural field. Neural points serve solely as spatial carriers of neural features in CP-SLAM, which are different from traditional keypoints and descriptors, similar to feature grid nodes in NICE-SLAM. In CP-SLAM, we do not perform inter-frame feature matching, so there is no discontinuity in tracking and mapping due to feature locations. We provide a detailed video in our supplementary material, which dynamically shows NERF-based tracking and mapping.\n\n**6. This paper should consider taking Nope-Nerf as a comparison. (Q2)**\n\nNope-nerf was not public before our paper was submitted. Furthermore, it is worth noting that Nope-nerf is more biased towards Structure-from Motion (SFM) rather than SLAM. It has a different background setting from our CP-SLAM, thus we didn't consider it as a comparable method.\n\n**7. Contributions of this paper are more biased towards engineering. (Q3)** \n\nWe aim to build a complete collaborative NERF-based SLAM system, focusing more on system-level development. Such work is missing in the field of NERF-based SLAM. CP-SLAM is the first collaborative neural implicit SLAM system, consisting of neural point-based odometry, loop detection, sub-map fusion, and global pose graph optimization. In collaborative NERF-based SLAM, in addition to the lack of appropriate neural field representation, it is not feasible to simply stack and cascade traditional modules in a plug-and-play manner. Consequently, we have introduced technical enhancements to both the neural field representation and the pipeline of CP-SLAM, such as innovatively proposing a keyframe-centric neural point field and a novel learning framework to maintain global consistency. Overall, we have made significant scientific contributions to the up-to-date NERF-based SLAM field as Reviewer Zpdr pointed out. Similar to Droid-SLAM [Teed et al. NeurIPS 2021], we also committed to a system-level development and introducing new learning-based neural implicit techniques into SLAM.\n"}]

Author rebuttal:
We thank all the reviewers for their valuable comments and appreciate the positive comments like “the contributions to the pipeline are significant” **(Reviewer Zpdr)**, "CP-SLAM introduces a novel neural point-based representation, enabling dense collaborative neural SLAM." **(Reviewer 1EKn)**, "The federated learning strategy improves the neural rendering performance. " **(Reviewer q5H1)**, "The use of point based features nice connects implicit mapping with the ability to do efficient map refinement" **(Reviewer zwcP)**, "This is the first time I saw a full SLAM system in the context of neural implicit SLAM." **(Reviewer 46C6)**, "Improved Performance in both camera tracking and mapping." **(Reviewer jr2H)**.  In this global response, we address the common concerns regarding our experiment and will separately address specific comments raised. 

- **Comparison with additional traditional methods:** In the multi-agent experiment, we added two traditional baselines including ORB-SLAM3 [Campos et al. 2021]  and Swarm-SLAM [Lajoie et al. 2023] as suggested for a more comprehensive comparison. For the single-agent experiment, we compared with ORB-SLAM3. The results are listed in **Table G1** and **G2**. We can observe that our method still performs the best compared to these methods. It is worth noting that ORB-SLAM3 is not a collaborative SLAM system. It lacks the capability to process multiple sequences at the same time, thus unable to overcome the efficiency bottleneck in scene exploration. 
- **Real-world experiment:** We have experimented on the TUM-RGBD real-world dataset. From **Table G3**, we can see that our CP-SLAM has also achieved state-of-the-art performance in the real-world setting, and the loop detection and pose graph optimization are effective for the real-world scene. The most recently proposed SOTA methods, i.e., Co-SLAM [Wang et al. 2023] and ESLAM [Johari et al. 2023], are evaluated for comparison in the TUM-RGBD experiment.

|   Method   |    Part     |    Apartment-1     |    Apartment-2     |    Apartment-0     | Office-0-C(SingleRoom) |
| :--------: | :---------: | :----------------: | :----------------: | :----------------: | :--------------------: |
|  CCM-SLAM  |  **Part1**  |   2.12/1.94/1.74   | **0.51/0.45/0.40** |       -/-/-        |     9.84/8.23/6.41     |
| ORB-SLAM3  |             |   4.93/4.65/5.01   |   1.35/1.05/0.65   |   0.67/0.58/0.47   |     0.66/0.62/0.62     |
| Swarm-SLAM |             |   4.62/4.17/3.90   |   2.69/2.48/2.34   |   1.61/1.33/1.09   |     1.07/0.96/0.98     |
| Ours(w/o)  |             |   1.15/0.99/0.88   |   1.45/1.34/1.36   | 0.70/0.48/**0.27** |     0.71/0.62/0.67     |
|  Ours(w/)  |             | **1.11/0.95/0.81** |   1.41/1.30/1.36   | **0.62/0.47**/0.30 |   **0.50/0.46/0.55**   |
|  CCM-SLAM  |  **Part2**  |   9.31/6.36/5.57   | **0.48/0.43/0.38** |       -/-/-        |   0.76/**0.36/0.16**   |
| ORB-SLAM3  |             |   4.93/4.04/3.80   |   1.36/1.24/1.11   |   1.46/1.11/0.79   |   **0.54**/0.49/0.47   |
| Swarm-SLAM |             |   6.50/5.27/4.39   |   8.53/7.59/7.10   |   1.98/1.48/0.94   |     1.76/1.55/1.83     |
| Ours(w/o)  |             |   2.12/2.05/2.23   |   2.54/2.45/2.60   |   1.61/1.55/1.70   |     1.02/1.03/0.99     |
|  Ours(w/)  |             | **1.72/1.61/1.46** |   2.41/2.33/2.44   | **1.28/1.17/1.37** |     0.79/0.74/0.70     |
|  CCM-SLAM  | **Average** |   5.71/4.15/3.66   | **0.49/0.44/0.39** |       -/-/-        |     5.30/4.29/3.29     |
| ORB-SLAM3  |             |   4.93/4.35/4.41   |   1.36/1.15/0.88   | 1.07/0.85/**0.63** |   **0.60/0.56/0.55**   |
| Swarm-SLAM |             |   5.56/4.72/4.15   |   5.61/5.04/4.72   |   1.80/1.41/1.02   |     1.42/1.26/1.41     |
| Ours(w/o)  |             |   1.64/1.52/1.56   |   2.00/1.90/1.98   |   1.16/1.02/0.99   |     0.86/0.81/0.83     |
|  Ours(w/)  |             | **1.42/1.28/1.14** |   1.91/1.82/1.90   | **0.95/0.82**/0.84 |     0.65/0.60/0.63     |

**Table G1: Two-agent Tracking Performance.  ATE RMSE/Mean/Median [cm] are reported.**

|   Method    |    Room-0-loop     |    Room-1-loop     |   Office-0-loop    |   Office-3-loop    |      Average       |
| :---------: | :----------------: | :----------------: | :----------------: | :----------------: | :----------------: |
|  NICE-SLAM  |   1.27/1.15/1.09   |   1.74/1.61/1.66   |   2.27/1.91/1.82   |   3.19/2.77/2.28   |   2.12/1.86/1.71   |
| Vox-Fusion  |   0.82/0.77/0.78   |   1.35/1.30/1.25   |   0.99/0.94/0.95   |   0.82/0.74/0.73   |   0.99/0.94/0.93   |
|  ORB-SLAM3  |   0.54/0.52/0.53   | **0.21/0.19/0.19** |   0.58/0.51/0.52   |   0.89/0.80/0.84   |   0.56/0.51/0.52   |
| Ours(w$/$o) |   0.61/0.56/0.54   |   0.51/0.48/0.52   |   0.67/0.63/0.67   | 0.38/0.32/**0.27** |   0.54/0.50/0.50   |
| Ours(w$/$)  | **0.48/0.44/0.43** |   0.44/0.40/0.46   | **0.56/0.53/0.56** | **0.37/0.31/0.27** | **0.46/0.42/0.43** |

**Table G2: Single-agent Tracking Performance.  ATE RMSE/Mean/Median [cm]  are reported.**

|  Method   | fr1-desk (w/o loop) | fr2-xyz (w/o loop) | fr3-office (w/ loop) |      Average       |
| :-------: | :-----------------: | :----------------: | :------------------: | :----------------: |
|  Co-SLAM  |   7.10/6.83/6.79    |   4.05/3.76/3.47   |    5.58/5.06/4.57    |   5.58/5.22/4.95   |
|   ESLAM   | **6.81/6.56/6.88**  |   Fail/Fail/Fail   |    4.23/3.91/3.73    |         -          |
| Ours |   7.84/7.34/7.12    | **3.93/3.50/3.29** |     **3.84/3.47/3.39**    | **5.20/4.77/4.60** |

**Table G3: Single-agent Tracking Performance on the TUM-RGBD dataset.** In the real-world experiment, we can find that CP-SLAM still performs the best. Besides, ESLAM fails in the fr2-xyz because of OOM (out of memory). '-' indicates that metrics cannot be evaluated due to ESLAM failures.

Meta-review: The paper presents CP-SLAM, a collaborative implicit neural SLAM system. The authors propose a neural point based 3D scene representation in which each point maintains a learnable neural feature for scene encoding and is associated with a certain keyframe. The authors also propose a distributed-to-centralized learning strategy to improve consistency and cooperation. A  global optimization framework is also proposed to improve the system's accuracy.

The paper received five “Weak accept” ratings and one “Borderline accept” rating.

Reviewer q5H1 gave “Weak Accept”. Reviewer q5H1 thinks the federated learning strategy improves the neural rendering performance. And, Reviewer q5H1 thinks some features are easily adjusted by encoding scene geometry and appearance with 3D points. But also, Reviewer q5H1 wants to see evaluations on more datasets. The authors gave their rebuttals. After reading them, Reviewer q5H1 had no further questions.

Reviewer jr2H gave “Weak Accept”. Reviewer jr2H thinks a neural point cloud representation for scene encoding in the proposed framework is a significant contribution. Reviewer jr2H thinks the proposed system can handle various challenges typically encountered in SLAM tasks. And, Reviewer jr2H thinks the incorporation of a distributed-to-centralized learning strategy is an effective solution for improving consistency and cooperation in collaborative implicit SLAM. But also, Reviewer jr2H wants to see more experiments. The authors gave their rebuttals. After reading them, Reviewer jr2H had no further questions.

Reviewer 46C6 gave “Weak Accept”. Reviewer 46C6 thinks this paper proposes the first full SLAM system in the context of neural implicit SLAM. Reviewer 46C6 thinks the system is well engineered, putting many components together and works effectively well. Reviewer 46C6 thinks the system achieves competitive tracking and mapping results compared with previous collaborative and neural SLAM methods. But also, Reviewer 46C6 thinks the novelty of this paper is not enough. The authors gave their rebuttals. After reading them, Reviewer 46C6 had no further questions.

Reviewer zwcP gave “Weak Accept”. Reviewer zwcP thinks the use of point based features nicely connects the generalization and compression of implicit mapping with the ability to do efficient map refinement. Reviewer zwcP thinks the results on tracking and reconstruction show a large improvement compared to existing works. But also, Reviewer zwcP thinks the authors should further clarify some concepts and should include more experimental results. The authors gave their rebuttals. After reading them, Reviewer zwcP had no further questions.

Reviewer 1EKn gave “Weak Accept”. Reviewer 1EKn thinks this paper introduces a novel neural point-based representation, enabling dense collaborative neural SLAM. Reviewer 1EKn thinks the system achieves state-of-the-art performance. Reviewer 1EKn thinks the implementation details are enough. But also, Reviewer 1EKn thinks the performance of the system is limited by its GPU resource demand. The authors gave their rebuttals. After reading them, Reviewer 1EKn had no further questions.

Reviewer Zpdr gave “Borderline accept”. Reviewer Zpdr thinks the paper is well-written, easy to follow, and well-organized. Reviewer Zpdr thinks the contributions are significant. But also, Reviewer Zpdr thinks the novelty of this paper is limited. Reviewer Zpdr wants to see more experimental results. The authors gave their rebuttals. After reading them, Reviewer Zpdr had no further questions.

In summary, the theoretical novelty is not enough and more contribution is on a system-level development.  Please make revisions and address all other questions raised by the reviewers. Then, the paper can be accepted by NeurIPS.
Decision: Accept (poster)