Q-SLAM: Quadric Representations for Monocular SLAM

Authors: chensheng  peng
chenfeng  xu
yue  wang
mingyu  ding
heng  yang
masayoshi  tomizuka
kurt  keutzer
marco  pavone
wei  zhan
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-22 09:52:39

Abstract: In this paper, we reimagine volumetric representations through the lens of quadrics. We posit that rigid scene components can be effectively decomposed into quadric surfaces. Leveraging this assumption, we reshape the volumetric representations with million of cubes by several quadric planes, which results in more accurate and efficient modeling of 3D scenes in SLAM contexts. First, we use the quadric assumption to rectify noisy depth estimations from RGB inputs. This step significantly improves depth estimation accuracy, and allows us to efficiently sample ray points around quadric planes instead of the entire volume space in previous NeRF-SLAM systems. Second, we introduce a novel quadric-decomposed transformer to aggregate information across quadrics. The quadric semantics are not only explicitly used for depth correction and scene decomposition, but also serve as an implicit supervision signal for the mapping network. Through rigorous experimental evaluation, our method exhibits superior performance over other approaches relying on estimated depth, and achieves comparable accuracy to methods utilizing ground truth depth on both synthetic and real-world datasets.

Main paper: 
 introduction: 
 Recent years have seen a renaissance of simultaneous localization and mapping (SLAM), thanks to the advances in learning-based localization methods [1, 2, 3] and neural radiance fields (NeRFs) [4, 5, 6] . In its basic incarnation, SLAM involves estimating per-frame camera poses and reconstructing 3D scenes from visual inputs. The key challenging of monocular SLAM lies in the accurate 3D scene geometry modeling from visual inputs.To enable high-quality 3D scene reconstruction, recent SLAM approaches resort to NeRFs and Gaussians as key map representations. Specifically, NeRFs [7] , employing neural networks i.e. MLPs, are recognized for their ability to achieve photo-realistic novel-view synthesis, as demonstrated in studies such as iMAP [5] , Nice-SLAM [8] . Conversely, 3D Gaussian Splatting (3DGS) [9] relies on dense Gaussian representations to learn scene geometries effectively. Despite achieving photo-realistic novel-view synthesis, NeRFs and Gaussians have a few drawbacks when applied in SLAM systems. On the one hand, MLP-based NeRFs are slow to train while achieving significant memory efficiency. On the other hand, 3D Gaussian Splatting offers near-real-time training capabilities but suffer from large memory consumption and sensitivity to inaccurate camera pose estimation. Additionally, both methods focus on dense high-quality reconstruction, demanding substantial representation efforts to capture fine-grained details.Addressing these challenges necessitates the design of map representations that are compatible with both learning-based localization front-end and NeRF(GS)-based reconstruction module. Therefore, to enhance scene geometry representation in SLAM, we introduce a novel method, termed Q-SLAM. As illustrated in Figure 1 , we integrate the proposed quadric representations across the SLAM pipeline in several key areas.(1) The well-established front-end tracking module can take RGB stream as input and offer a rough depth estimation while estimating camera poses. Nevertheless, this depth estimation encounters significant challenges, particularly at edges and in texture-less regions, where accuracy tends to degrade. To address these issues, we introduce a quadric-based depth correction module, leveraging quadric surfaces to refine the depth estimation, thus improving overall scene reconstruction accuracy, especially in complex and challenging environments.(2) In the mapping module, departing from previous NeRF-based SLAM approaches like [4, 8] , which rely on volumetric representations, our method transforms densely volumetric scenes -comprising millions of cubes (e.g., 300 × 300 × 300 voxels) -into a more manageable set of quadric surfaces (50-100 quadric surfaces per keyframe). This approach maintains the integrity of the 3D scene descriptions while significantly reducing complexity, by sampling around the quadric surfaces instead of the whole 3D space.(3) During the rendering phase, we develop a quadric-ray transformer. We employ importancesampling based on the quadric-rectified depth values, and we emphasize that sampled points on a quadric naturally belong to the same instance, which facilitates the modeling of their interrelationships during rendering, akin to the method in [10, 11] . We first perform feature interaction between points along a ray to aggregate information and then model the relationships between these surface points across rays using a transformer.(4) Notably, we extend the utility of quadric semantics beyond mere depth correction and scene decomposition. They are also used as a supervision signal throughout the optimization of the NeRF network to learn robust and accurate features. With the integration of quadrics into the pipeline, we also consider the estimated pose as a learnable parameter in our mapping phase. This introduces a differentiable optimization process to refine the pose estimation, enhancing overall system accuracy. 
dense visual slam: 
 Neural Radiance Fields (NeRF) [12] have exhibited notable efficacy in diverse applications, including view synthesis [12, 10, 13] and 3D reconstruction [14, 15] . RGBD SLAM: Innovative contributions such as iMAP [5] and Nice-SLAM [8] have pioneered the integration of NeRF into SLAM systems, demonstrating commendable tracking and mapping performance for indoor scenes. Co-SLAM [1] adopts a multi-resolution hash-grid representation for the scene to expedite convergence. E-SLAM [16] integrates the multi-scale axis-aligned perpendicular feature planes for efficient reconstruction of 3D scenes. By anchoring neural scene features in a point cloud generated iteratively in an input-driven manner, Point-SLAM [17] optimizes runtime and memory usage while maintain-ing fine detail resolution. Besides using NeRF as the map representation, recent SLAM systems also resort to 3D Gaussian Splatting (3DGS) [9] due to its explicit representation property and real-time rendering performance. SplaTAM [18] pioneers in replacing NeRF with 3DGS and has achieved promising results in both tracking and mapping. GS-SLAM [19] further proposes a coarse-to-fine technique to select reliable 3D Gaussian representations for camera pose optimization effectively. However, these GS-based approaches have one drawbacks in common, much higher requirements for memory usage. All these approaches above rely on ground truth depth map for supervision of NeRF (or 3DGS) training, which is hard to obtain in real-world environment, thereby limiting their applicability in robotics. Conversely, Monocular SLAM method emerges, with only RGB images as input. In particular, NeRF-SLAM [4] integrates DROID-SLAM [20] for pose estimation and depth prediction and use Instant-NGP [21] to fit a NeRF. GO-SLAM [2] globally optimize poses and 3D reconstruction. However, the performance of these methods in 3D reconstruction is impeded by the inherent limitations of noisy predicted depth maps.Volumetric Representations are predominantly utilized in Neural Radiance Fields (NeRFs) [22, 23, 12] , and in 3D perception [24, 25, 26] . These representations encapsulate a 3D scene through a volumetric approach, where each volume element embodies occupancy probabilities or task-specific features. However, deriving volumetric representations solely from visual inputs presents significant challenges, particularly in terms of geometric estimation, leading to ambiguous and noisy outputs. Point-Cloud Representation offers a sparser alternative [27] . Point-SLAM [17] utilize a dynamic point density strategy to reduce computational and memory usage. Yet sparse point-clouds struggle to depict scenes comprehensively, while dense point-clouds encounter similar complications to volumetric representations, as previously mentioned. Consequently, few vision-based SLAM systems adopt point-based representations. Additionally, primitives like Planes [28] and Quadric-Meshes [29, 30, 31] have been proposed for scene representation. QuadricSLAM [32] estimates 3D quadric surfaces from 2D multi-view images and uses them as landmark representations for camera tracking in the SLAM pipeline. Dense Planar SLAM [33] uses bounded planes and surfels extracted from depth images to build a real-time SLAM system, but ignore the fine-grained reconstruction of 3D scenes. Point-plane SLAM [34] exploits the constraints of plane edges, which are predominant features that are less affected by measurement noise. ManhattanSLAM [35] utilizes the indoor structural information to estimate camera poses based on the Manhattan World (MW) assumption. However, these are typically employed for regularization during optimization or are challenging to optimize, limiting their applicability in visual SLAM. Drawing inspiration from quadric representations in LiDAR SLAM [3] , we introduce quadric representations in visual monocular SLAM. 
overview: 
 Our framework, Q-SLAM, as illustrated in Fig. 2 , takes monocular RGB sequences as input. Initially, Droid-SLAM [20] predicts rough depth maps and initial camera poses from these inputs. Concurrently, a pretrained segmentation network is employed to estimate segmentation masks from these images. These masks are then utilized by the quadric-based depth correction module to refine the noisy depth maps, yielding more accurate corrected depth maps. Alongside the corresponding RGB images, camera poses, and segmentation results, they are inputted into the NeRF network.For NeRF optimization, the RGB images, corrected depth and segmentation masks serve as supervision signals. To capture the semantic relationship across quadric surfaces, we further propose a quadric ray transformer, enabling effective feature interaction within and across the sampled rays. During the mapping process, both the camera poses G = g t and the 3D scene representation parameters Θ are jointly optimized to enhance tracking and mapping accuracy. With the learned NeRF parameters, we can render RGB images, depth maps, and semantic maps for novel views, which can then be utilized to reconstruct the 3D mesh using TSDF-Fusion [36] .NeRF OptimizationFigure 2 : Overview of our proposed method. From the input RGB sequences, we can predict depth map, camera pose and segmentation mask. Subsequently, the initially estimated depth undergoes correction based on the quadric assumption. Along with the segmentation mask, camera poses, and image frames, the corrected depth are used for optimization of NeRF network. During the 3D reconstruction process, our proposed quadric ray transformer leverages the quadric priors effectively.Through the inverse re-projection with the pixel locations (u, v) and the depth values d, a set of 3D points p can be obtained for each segmented patch, where the depth value directly serves as the z coordinate and K is the calibration matrix:[X, Y, Z] T = d • (K -1 [u, v, 1] T )To represent the points of each segmented patch as quadric surfaces, we define the quadric implicit function as follows:EQUATIONwhere x = [x, y, z] T is the linear term and q = x 2 , y 2 , z 2 , xy, yz, xz T is the quadric term calculated from x, and C q , C l and c are the coefficients to be fitted. Following Narunas et al. [37] , we define the cost function for the least-square fitting as the sum of squared distances between the quadric surface and the actual N points to be fitted:EQUATIONMinimizing over Eq. 2, we obtain the quadric coefficients. We preserve the patches with fitting error below a predefined threshold, which implies a good fitting surface for the following depth correction. By substituting (X, Y ) back into f (C, x), an equation involving z with other variables as constants is obtained. Solving this equation yields new Z values, representing the rectified depth values. The established quadric surface model proves effective in mitigating noise impact on depth information, ultimately improving the fidelity of the reconstructed spatial data. Fig. 3 illustrates the effectiveness of our proposed quadratically rectified approach. The original patch exhibits drifting points, particularly along the boundary where significant depth value changes occur. However, the proposed approach rectifies these points, fitting them to the quadric surface.The incorporation of additional constraints imposed by quadric surfaces contributes to a more precise depth prediction, thereby enhancing tracking and mapping accuracy.Intra-Ray Transformer: Drawing inspiration from IBRNet [10] , our approach employs an attention mechanism along the ray to model relationships between sampled points. (Fig. 4 ) , where B is the number of sampled rays, we query the volume density features f σ ∈ R B×(Ns+N d )×D and feed them into the intra-ray transformer. Processing density features instead of color features is a wise choice as they solely depend on xyz positions, simplifying feature aggregation based on the attention mechanism. Additionally, the smaller number of feature channels in density does not significantly increase the computational burden, resulting in an efficient representation for the application of SLAM.InitiallyThe density features of each point along the ray are updated using self-attention:EQUATIONwhere δ p is the positional encoding in attention mechanism. This self-attention mechanism facilitates feature aggregation along a ray, enabling the capture of more information from the surface and nearby points for more precise surface reconstruction.Inter-Ray Transformer: Intuitively, points belonging to the same semantic quadric are more likely to exhibit similar textural and spatial features, while rays sampled from different quadric surfaces might differ from each other in terms of textures and geometries. Therefore, we further propose the inter-ray transformer to capture the relationship across rays.The inter-ray transformer operates on the updated density features from intra-ray transformer. First, the updated density features f ′ σ are concatenated with the semantic features f s . Followed by a fusion network, new density features with semantic priors, f ′′ σ ∈ R B×N ×D , can be obtained.f ′′ σ = MLP f usion f ′ σ ⊕ f s (4)As shown in Fig. 4 , the inter-ray transformer then takes the transposed density features, f ′′ T σ ∈ R N ×B×D , as input, where attention map is calculated across different rays along the B dimension:EQUATIONwhere δ ′ p is the positional encoding in attention mechanism. The inter-ray transformer further facilitates feature interaction across rays. This process allows surface points to capture more inter-ray information from other rays from the same quadric surface. Together with the intra-ray transformer, our proposed network achieves feature aggregation across a broad range of ray points, enhancing rendering accuracy by incorporating quadric priors and additional spatial information.𝑓𝑓 𝜎𝜎 from other samples on the same rayInter-ray transformer During mapping, we map the 3D coordinates x, and viewing directions d to volume density, color and semantic logits (σ, c, s). Subsequently, volume rendering is utilized to reconstruct color Ĉ, depth D, and semantic map Ŝ. For points {p i |i = 1, • • • , M } on a fitted quadric surface, the color and depth loss is calculated as follows:EQUATIONHere, the fitting error ε serves as an uncertainty penalty. In cases where the fitting is not perfect, the quadric plane is less likely to exhibit similar texture and spatial features, resulting in reduced usefulness of the quadric transformer. Consequently, such patches are down-weighted by incorporating the fitting error ε. Following [38] , we use a multi-class cross-entropy loss as the semantic loss L s .Hence the rendering loss is a weighted sum of L c , L d and L s with hyperparameters λ 1 , λ 2 .EQUATIONFor joint optimization, the camera pose G = {g t }, a 4 × 4 transformation matrix, is converted to quaternion (4 × 1) and translation (3 × 1) vector, which are then taken as trainable parameters to the network, optimized together with the NeRF parameters while minimizing the rendering loss. In this way, the camera poses G and network parameters Θ can be jointly optimized. 
experimental setup: 
 Dataset. Q-SLAM is evaluated on a variety of datasets, including Replica [39] , ScanNet [40] , and KITTI MOT dataset [41] . For evaluation of the reconstruction quality, we test our method on 8 synthetic scenes from Replica, which provides high-quality synthetic scenes. Following GO-SLAM [2] , we evaluate the tracking accuracy on ScanNet dataset which offers extensively annotated RGB-D scans of real-world scenarios, encompassing challenging short and long trajectories. KITTI MOT dataset is an outdoor autonomous driving dataset with dynamic objects, such as cars. For camera tracking assessment, our approach is evaluated under two distinct modes: one utilizing ground truth and the other utilizing estimated depth from monocular images as inputs.Metrics. We evaluate tracking accuracy by aligning the estimated trajectory with the ground truth trajectory and computing the Root Mean Square Error (RMSE) of the Absolute Trajectory Error (ATE). This metric quantifies the Euclidean distance between the estimated pose and the corresponding ground truth pose. Following NeRF-SLAM [4] , we utilize Peak Signal-to-Noise Ratio (PSNR), SSIM [42] , and LPIPS [43] for image rendering evaluation, which evaluate the similarity between the rendered images and the ground truth images, and Accuracy [cm], Completion [cm],Completion Ratio [%] for 3D reconstruction assessment, which measures the reconstruction quality by comparing the reconstructed mesh and the ground truth mesh. Implementation Details. All experiments are conducted on an NVIDIA A6000 GPU using Py-Torch 1.10.0. We use Adam as the optimizer with β 1 = 0.9 and β 2 = 0.999. The tracking module is adapted from Droid-SLAM [20] , and pretrained weights are utilized to estimate depths and poses. Our mapping backbone is adapted from Point-SLAM [17] , incorporating our proposed depth correction and quadric ray transformer. For faster image segmentation, we use the MobileSAM [47] to obtain segmentation masks from each input RGB frame. Diverging from methodologies where scene meshes are reconstructed using marching cubes on Signed Distance Function (SDF) values of queried points, Q-SLAM renders images and depths over the estimated camera trajectory and use TSDF-Fusion [36] for mesh construction with voxel size 1 cm. During the fitting process, only quadric surfaces with fitting error lower than the specified threshold undergo depth correction; otherwise, the uncorrected predicted depth is used for the supervision of NeRF training. For joint optimization, the scene representation parameters Θ are optimized for five steps, and the accumulated losses are then utilized to update camera pose G.Replica dataset. We evaluate on the 8 scenes as Nice-SLAM and iMAP. As shown in Tab. 1, the results demonstrate comparable performance across almost all scenes in terms of rendering and reconstruction metrics. Notably, our approach exhibits performance on par with SLAM systems that utilize ground truth depth for supervision.ScanNet dataset. For a comprehensive comparison, we further evaluate our SLAM system on Scan-Net [40] dataset, encompassing both long and short sequences. As depicted in Tab. 2, our method outperforms other approaches in both monocular and RGBD setting, with a more pronounced superiority evident in the case of monocular inputs. Runtime and Memory Usage. In Table 3 , we also report the runtime and memory usage on the Replica dataset. The tracking and mapping time is reported per frame. It can be observed that our method can achieve comparable speed with Vox-Fusion [44] , but providing much higher rendering quality as shown in Tab 1. We also report the runtime of individual breakdown components in Tab 5.Depth Correction. In Tab. 4 (a), we compare the performance with or without the proposed quadric depth correction module. It can be observed that the depth correction process enhances the accuracy of the predicted depth map by introducing additional surface constraints, rectifying drifting points that deviate from the surfaces. This improvement positively impacts both tracking and mapping performance, especially for the Depth L1 metric. Joint Optimization. For experiments without joint optimization, the estimated camera poses are not optimized using the rendering loss during the mapping process. It can be observed from Tab. 4 (c) that joint optimization of camera poses and 3D reconstruction demonstrates a modest improvement in both tracking and mapping accuracy. We believe that the limited improvement stems from the fact that joint optimization is solely conducted on keyframes during the mapping process. Given that keyframes constitute only a small portion of the entire sequence, while the evaluation of camera poses spans all frames, this potentially constrains the impact of joint optimization.Semantic Supervision. With an additional semantic head, our model not only renders color and depth but also generates the semantic map. It's intuitive to utilize segmentation results from previous modules as a supervisory signal, enriching the available information. As shown in Table 4 (d), incorporating semantic supervision can significantly enhance performance. 
conclusion: 
 In this paper, we introduce quadratic surfaces as the map representation for SLAM. Based on segmentation results from RGB images, the roughly estimated depth values can be corrected by incorporating additional surface constraints. We utilize the rectified depths and quadric semantics as a prior for sampling points along the ray, significantly reducing the required number of samples to achieve comparable results, thus alleviating the computational burden. Additionally, we employ a novel quadric-ray transformer model to capture interrelations across different samples along the ray within the constraints of quadric surfaces. Furthermore, we propose an end-to-end joint optimization approach for pose estimation and 3D reconstruction. Sufficient experiments on various datasets demonstrate the effectiveness of our proposed method in terms of novel-view synthesis, depth estimation, and camera tracking.Limitations. The heavy computational cost of our method compared to traditional SLAM systems and the difficulty in scaling to unbounded outdoor scenarios. We also acknowledge that the quadric assumption of the surface is not ubiquitous, especially for complex structures in outdoor scenarios, such as trees. Additionally, the smoothing process heavily relies on the segmentation results. We plan to explore more efficient representations, such as gaussian splatting, in the future work. 
 
 Appendix: By setting ∇C c = 0 in Eq. 2 to obtain the optimal cEQUATIONThe cost function in Eq. 2 becomes:EQUATIONwhere q = 1 N N i=1 q i are the quadric term averaged on points in a patch, x = 1 N N i=1 x i is the linear term.The intermediate variables are defined as follows:EQUATIONBy substituting C * l back to Eq. 10, we can obtainEQUATIONMinimizing Eq. 13 over c q gives the eigenvector c * q of Ψ corresponding to the minimum eigenvalue, and c * l can be solved from Eq. 12, and c * from Eq. 9. As defined by Taubin et al. [49] , the distance from a point x to a quadric surface f is:EQUATIONFor every fitted patch, we calculate the average distance between the original points to the fitted surface as the fitting error. Those patches with error exceeding the given threshold will be discarded.We only preserve the patches with relatively small fitting error, which implies a good fitting surface for the following depth correction. While the static assumption applied in existing NeRF SLAM methods is suitable for indoor environments, it becomes inadequate for outdoor scenarios with moving objects, particularly in autonomous Figure 7 : Qualitative reconstruction results on KITTI MOT dataset. We compare our solution with recent SOTA NeRF-SLAM systems Co-SLAM [1] and GO-SLAM [2] .driving contexts, limiting their applicability. We propose to extend our method to outdoor scenarios, which are rarely explored in previous methods [8, 17, 16, 18] . By incorporating an additional dynamic branch, we can effectively model the reconstruction of moving objects using semantic priors. Unlike the static branch, which outputs (σ s , c s ), the dynamic branch produces outputs (σ d , c d ) that vary with time t. The time embedding, derived from t is integrated into the dynamic branch along with other inputs. The final density σ and color c for rendering are calculated as follows:EQUATIONwhere x and d represents the 3D coordinates and viewing direction respectively, and t is the timestamp. The subscript s and d stand for static and dynamic branch respectively, and the outputs of dynamic branch depend on time, while the static branch does not.The color Ĉ and depth D are rendered as follows:EQUATIONEQUATIONWe do not incorporate the semantic head for outdoor scenes, because there are much more classes of objects compared to indoor scenes. To supervise the training on dynamic objects, we generate the static masks following [50] , and apply a regularization loss of dynamic branch on the static regions.EQUATIONWe conducted experiments on KITTI dataset [41] , including outdoor scenes with moving cars. The qualitative results are presented in Fig. 7 .The data of other NeRF-SLAM methods in Tab. 1 is sourced from Nicer-SLAM [46] , and the geometric reconstruction results (Acc., Comp., etc.) of SplaTAM [18] comes from RTG-SLAM since the original paper does not report these metrics. The results of other method in Tab. 2 are mainly taken from GO-SLAM [2] .Q-SLAM is evaluated on a variety of datasets, including Replica [39] , ScanNet [40] , and TUM RGB-D [51] dataset. For evaluation of reconstruction quality, we test our method on 8 synthetic scenes from Replica, which provides high-quality synthetic scenes, akin to the evaluation framework adopted by NeRF-SLAM [4] . Following GO-SLAM [2] , we evaluate the tracking accuracy on ScanNet dataset which offers extensively annotated RGB-D scans of real-world scenarios, encompassing challenging short and long trajectories. Following Nice-SLAM [8] , we also evaluate on various scenes on indoor TUM RGB-D dataset, with ground truth poses provided by a motion capture system. For camera tracking assessment, our approach is tested under two distinct modes: one utilizing ground truth and the other utilizing estimated depth from monocular images as inputs.The batch size of sampled rays to NeRF network is 8192.We evaluate tracking accuracy by aligning the estimated trajectory with the ground truth trajectory and computing the Root Mean Square Error (RMSE) of the Absolute Trajectory Error (ATE). This metric quantifies the Euclidean distance between the estimated pose and the corresponding ground truth pose. In line with the evaluation approach of NeRF-SLAM [4] , we utilize Peak Signal-to-Noise Ratio (PSNR), SSIM [42] , and LPIPS [43] for image rendering evaluation, and Accuracy [cm], Completion [cm], Completion Ratio [%] for 3D reconstruction assessment.• Absolute Trajectory Error (ATE) (cm) ↓: Evaluates trajectory estimation accuracy by measuring the average Euclidean translation distance between corresponding poses in estimated and ground truth trajectories, often reported in terms of Root Mean Square Error (RMSE).• Peak Signal to Noise Ratio (PSNR) ↑: Measures image quality by evaluating the ratio between the maximum pixel value and the root mean squared error, usually expressed in terms of the logarithmic decibel scale.• Structural Similarity Index Measure (SSIM) ↑: Assesses image quality by examining the similarities in luminance, contrast, and structural information among patches of pixels.• Learned Perceptual Image Patch Similarity (LPIPS) ↓: Utilizes learned convolutional features to assess image quality based on feature map mean squared error across layers.• Accuracy (cm) ↓: Computes the average distance between sampled points from the reconstructed mesh and the nearest ground-truth point.• Completion (cm) ↓: Measures the average distance between sampled points from the ground-truth mesh and the nearest reconstructed.• Completion Ratio (%) ↑: the percentage of points in the reconstructed mesh with Completion under 5 cm.All experiments are conducted on NVIDIA A6000 GPU with PyTorch 1.10.0. We use Adam as our optimizer with β 1 = 0.9, β 2 = 0.999. The tracking backbone is Droid-SLAM, where we use the pretrained weights to estimate depths and poses. We use Point-SLAM as the mapping backbone, equipped with our proposed depth correction and quadric transformer. The threshold for motion filter is 4.0 pixels, a tracked frame is considered as a keyframe only if the average optical flow is greater than the threshold. The window size for local bundle adjustment is 25. During the joint optimization process, camera poses are optimized for one epoch, and the NeRF network parameters are optimized for five epochs.For the segmentation network, we use Segment Anything Model (SAM) [52] and MobileSAM [47] for faster inference, an off-the-shelf network to produce the mask for quadric fitting. To prevent the negative effect of outliers, quadric fitting only applies to segments with area larger than 200 pixels.During the fitting process, we calculate the coefficient of determination to evaluate the fitting performance. Let z i be the predicted depth value and f i be the corresponding corrected depth. The 8 : Qualitative reconstruction results on Replica dataset. We compare our solution with recent SOTA SLAM systems Co-SLAM [1] and GO-SLAM [2] . Our method can recover better texture features, especially on the boundary of instances.We provide several selected visualization results for depth correction as shown in Fig. 9 . Benefiting from the segmentation mask, the depth correction improves the sharpness of the boundary of objects. https://deepdrive.berkeley.edu 


Decision: Although the paper is recommended for acceptance, it is strongly encouraged that the authors consider changing the presentation to a paper that focuses on 3D reconstruction on the camera-ready version (including the title). 

Note that the novelty of the quadric representation is clear for 3D reconstruction, while the SLAM part uses existing approaches. 

For the rest, the authors successfully addressed most of the reviewers' concerns. Please include all the required modifications in the camera-ready version.

Summary of Strengths:
* Interesting integration of quadrics 
* Good results compared with the state-of-the-art
* Quadrics are used to regularize the depth

Summary of Weaknesses:
* Limitations are not well addressed
* The paper is about the scene representation not really about SLAM
* Clarifications and more details are needed
* Missing references
* Missing evaluation of quadrics are used to regularize the depth

Reviews: review: title: title: Q-SLAM: Quadric Representations for Monocular SLAM
paper_summary: paper_summary: This paper shows how to impose quadric-constraint on scene segment in the context of 3D SfM (or SLAM). The core idea is to use an image segment (from a segmented network), fit a quadric to them and force the segment points ot the quadric. The corrected depth can then be used to support scene reconstruction with a NERF. A second contribution is to use a ray transformer to improve NERF performance.
main_review: main_review: This paper shows how to impose quadric-constraint on scene segment in the context of 3D SfM (or SLAM). The core idea is to use an image segment (from a segmented network), fit a quadric to them and force the segment points ot the quadric. The corrected depth can then be used to support scene reconstruction with a NERF. A second contribution is to use a ray transformer to improve NERF performance.  

The use of quadrics to approximate parts of the scene is very interesting. In a way the quadrics are used to regularize the depth. This regularization is claimed to improve the precision of the depth reconstruction but this is not specifically evaluated. Another point that would be worth discussion is the sensitivity of the fit to outliers: would it be helpful to run a RANSAC quadric fitting per segment, before projecting the segment points on the surface, or is it not necessary because NERF will handle it later ? 
When looking at the video, the resulting depth seems to be extremely smooth. This is a property of the scene but may not be true everywhere, specifically in scene segment containing vegetation for instance.

The second contribution is the use of a specific ray-transformer to improve NERF performance. The transformer itself seems to be independent of the quadric representation, but the papers shows that the quadric prior supports the sampling of the rays. 

Overall, the evaluation and comparisons with sota are by the book, with the tables as expected. An ablation study is provided but not a specific limitation section.

Although the approach is extremely interesting, the global impression is that the paper is trying to fit too much in the limited space, which results in a limited amount of details and insight.
questions: questions: The main questions to address would be the discussion on the robustness to outliers, and the dependence between the ray transformer model and the quadric sampling for NERF. Could these two components be evaluated independently in the ablation study?
limitations: limitations: 1
review_summary: review_summary: This paper proposes an interesting integration of priors in the Nerf-based SfM pipeline. This is relevant to robotics but mostly a computer vision contribution.
title: title: Good 3D reconstruction work with many concerns and details to be addressed. Additionally, it should not be presented as a SLAM Paper.
paper_summary: paper_summary: This paper presents a novel 3D reconstruction method using compact quadric representation with neural implicit representation.
main_review: main_review: **Strengths:**
1. The proposed method shows good results in 3D reconstruction and pose estimation, with comparable memory consumption and latency.
2. Overall, the paper is easy to follow, and the method has been tested in different environments, showing good generalization.

**Weaknesses:**
1. First of all, I strongly oppose the presentation of this paper as a SLAM work. The paper does not mention any of its pose estimation components: how its front-end tracking works, how it handles temporal data, or how the loop closure is implemented (if it claims to be a SLAM paper, instead of an odometry paper). I feel this paper mainly (if not only) focuses on the 3D reconstruction given posed RGB inputs.
2. Incomplete reference work: the related work mainly includes very recent neural implicit RGB-D SLAM, ignoring the conventional works on RGB-D SLAM and Scene Representations.
3. Too many details are missing in the paper. The initial depth prediction/estimation, segmentation, bundle adjustment, and camera tracking components are barely mentioned. Without these details, it is impossible to reproduce or even judge the quality of this work. These are also essential components of a SLAM system. If the pose estimation and optimization stages are not explained, there is no point in evaluating them.
4. There is no discussion of limitations or failure cases.
questions: questions: * The experiment hardware setup for training and the inference speed have been mentioned in the paper. Is the test platform for inference speed the same as the training platform? Referring to Table 3, on which test platform were the experiments conducted? Were the three methods all tested on the A6000 GPU? Additionally, there should be an analysis of its components (refer to Figure 6 in the appendix).

* How does this work compare to other quadric-based SLAM methods, especially the following:
  * Laidlow, Tristan, and Andrew J. Davison. "Simultaneous localisation and mapping with quadric surfaces." 2022 International Conference on 3D Vision (3DV). IEEE, 2022.

* Table 3: All the comparison methods require heavy GPU computation, which is impractical for some power-limited robot platforms. Can the authors also please compare some dense SLAM methods that have been widely adopted on mobile robot platforms:
  * Vespa, Emanuele, et al. "Efficient octree-based volumetric SLAM supporting signed-distance and occupancy mapping." IEEE Robotics and Automation Letters 3.2 (2018): 1144-1151.
  * Oleynikova, Helen, et al. "Voxblox: Incremental 3D Euclidean signed distance fields for on-board MAV planning." 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017.

* There needs to be a justification for why the optimized NeRF field needs an extra post-processing step to run TSDF-Fusion again. The NeRF field can be directly converted to 3D meshes.

* I am also a bit confused about Table 5 in the appendix. It reports that ORB-SLAM2 and ORB-SLAM3 failed on half of the TUM RGB-D datasets. This is confusing as I have personally tested ORB-SLAM on these scenes, and it definitely did not show such a high failure rate.

* How does the method scale to larger environments? Since it is based on quadric surface representation, I assume it should have better memory efficiency compared to volumetric representation. But this needs to be confirmed in the experiment.

* The quadric ray sampling adds some regularization, implying that points on the same quadric exhibit similar features. Will the quadric representation cause some loss of geometric details? Is it sensitive to segmentation errors? For example, in the ScanNet scene where objects are clustered, objects tend to be missed in the segmentation. Will the semantic error affect the geometry quality?

* Some notations in Section 3.3 are not explained.
limitations: limitations: 1
review_summary: review_summary: This work shows some advancements in 3D reconstruction, but too many details are missing for it to be considered a SLAM paper. I would strongly reject it as a SLAM paper unless the authors are willing to reformat it as a 3D reconstruction work. Maybe it is more suitable in a computer vision venue as a 3D vision paper.
title: title: The design of the SLAM system reads reasonable for scenes containing many simple surfaces. Some parts are somewhat unclear and could benefit from further clarification.
paper_summary: paper_summary: This paper proposes a framework that combines volume rendering with SLAM. Instead of representing the entire scene using an MLP like NeRF, the proposed method divides the space into millions of cubes and extracts quadric planes for environment representation. Specifically, raw RGB images are input into the pre-trained DROID-SLAM, which provides initial poses and depth maps. The RGB images are also fed into a pre-trained segmentation network to obtain segmentation masks. The predicted depth maps and segmentation masks are then fused under the quadric assumption to produce a refined depth map. Utilizing the refined depth map, the camera poses and NeRF parameters are optimized, with a transformer used to share information between sampled points along and between the rays.
main_review: main_review: and exploration of this field.

Major Concerns and Questions:

- The benefit of adopting a quadric function to correct depth noise for environments with simple surfaces is clear. However, have you tested the method on scenes where the object’s surface is not locally smooth, such as a durian or some other objects? Including this additional environment could provide readers with a better understanding of the proposed quadric representation's capabilities.

- The scene is initially divided into millions of cubes. Is this division solely for obtaining quadric surfaces? In other words, do you use a single MLP for the entire scene representation, similar to NeRF?

- Line 107 mentions that the tracking module predicts rough depth maps and initial camera poses. This was confusing until I read Section 4.1, which clarifies that the pre-trained DROID-SLAM is used as the tracking module. It would be helpful to mention this with proper citation around line 107 to make the paper easier to understand.

- What pre-trained segmentation network is used for obtaining the segmentation map? Please clarify this with proper citations. Additionally, how many labels does the segmentation map provide? Is it trained to identify flat or nearly-flat surfaces rather than objects, as shown in Figure 2? Providing more details would be beneficial.

- Regarding quadric depth correction, are the parameters estimated per depth? In other words, do you calculate the quadric parameters in the world frame and transform them into the camera local frame using the camera pose to get the depth map, or do you estimate the parameters for each local frame? It makes sense if the parameters are in the world frame, but based on Section 3.2, it seems they are estimated in the latter way.

- For sampling points, do you sample points along the ray, like NeRF, or do you use the depth map as a guide? Please provide more details.

- For joint optimization (lines 184-187), do you refine a single camera pose or jointly estimate the camera poses of multiple frames? The former method is more akin to visual odometry rather than SLAM.

- In Table 2, why is the performance of the proposed method on datasets 0059-0181 not as good as on the others? I am curious about what kind of environments or trajectories affect the performance. Even a preliminary explanation would be helpful.
questions: questions: Please refer to my review comments above.
limitations: limitations: 2
review_summary: review_summary: This paper enhances DROID-SLAM by incorporating quadric depth correction and the ray transformer. - Some parts of the paper are not very clear, and I would like to see the authors' response to these points. - The attached code is helpful in demonstrating and validating the proposed method.