Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper proposes a novel deep learning-based SLAM algorithm. The key technical contribution is to adopt a recurrently updated deep optical flow to serve as the estimation of correspondence and build a differentiable bundle adjustment solver module within the network. The proposed approach is solid and technically sound. It sets the new state-of-the-art and demonstrates exciting potential opportunities. The paper is clearly presented. I would strongly recommend accepting this paper. 

main_review: Pros:
* Very strong results. Significantly better than prior art on TartanAir, EuRoC, ETH3D, and TUM and close to being real-time. It's exciting to see such results on SLAM. 
* The approach is simple, elegant, and makes a lot of sense. I am particularly interested in the iterative learning of the residual flow. 
* Less hyper-parameter tuning and engineering trick than most of the existing visual SLAM approaches. 
* Easy to generalize to monocular depth, stereo, and RGB-d settings. It also demonstrates generalization capability across different datasets. 


Cons/ Detailed Comments:


Writing:
* I would not claim the DBA layer is new — as you pointed out in the paper and some other existing literature, very similar instantiation has been seen before. It won’t hurt even if this part is not entirely novel — this submission has a good and novel design of the entire SLAM approach. 
* There are not enough discussions to justify the technical choice and superior performance: 1) why rejection error is superior to the photometric ones as in indirect method; 2) why dense optical flow than sparse key feature correspondence? 3) why do you need the residual updates of the flow estimation? 
* Do you see any failure cases in some testing videos? Could you show some failure cases?
* Qualitative results are not sufficient for this paper. I would like to see visualizations of the flow confidence map, intermediate flow & depth results, iterative updates, before vs after local BA, and an online demo video in the final supplementary. 
* 

Approaches:
* It’s not clear to me how did you select pixels for the local bundle adjustment (all or prune some by the confidence weight?) and how dense you store the features for global BA. 
* Many papers have seen superior results using differentiable LM solver than GN solver, with a proper damping factor scheme. Have you compared both? 
* Do you need a good initial pose estimation for a new incoming frame at the inference?
* Initialization from the first 12 frames is interesting to me: how do you initialize your pose then? All zeros? Won’t they fall into some local optimal? If not, do you have some intuitions why? This is different from the conventional approach, where we might need a good init from homography or essential decomposition. 
* Have you double-checked whether Schur complement is indeed superior to other options, e.g., Cholesky in this case? Given the structure, Cholesky could be faster as well. 
* It seems you directly remove the keyframe, have you considered conducting marginalization using the scour complement? I would love to see whether it makes a difference here. 
* Is there any way for the network to decide on a tracking failure in some cases? This will be very useful in practice. Is the learned energy value a good indicator of success? 


Experiments:
* There isn’t much ablation on the tech choice, e.g., optical flow-based results vs. only using dense depth-based results; different options for local BA windows. 
* Following the above discussion on flow vs. feature matching, I am actually wondering how well superpoint+superglue + your proposed frontend DBA layer + your backend do. This would be a nice experiment. 
* The scale of the global bundle adjustment is still very big. How do you deal with it? Do you directly implement with PyTorch tensors? 
* Given the sparse block patterns, have you considered using a sparse tensor in the implementation to reduce memory footage and accelerate, like what has been done in ceres/gtsam? 
* Is it straightforward to integrate a loop-closure module in the backend? 


Relevant works:
* gradSLAM: Automagically differentiable SLAM, CVPR 2020: this is a similar end-to-end approach based on a differentiable bundle adjustment solver. That being said, both frontend and backend are different, and this submission’s result is stronger. 
* Deep rigid instance scene flow, CVPR 2019; and RAFT-3D in CVPR 2020: these approaches have a similar DBA-layer optimizing reprojection error. The correspondence is based on deep flow estimation, but this approach is two-frame and mainly focuses on instance-level 3D scene flow. Besides, it would help if you discussed VOLSOR and Flowfusion more explicitly in related work, as they are also deep optical flow-based slam and directly relevant. This helps better position the paper and acknowledge prior work.
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper learns to estimate the camera poses and depth by interlacing the residual flow field estimation and the geometric bundle adjustment iteratively. At each iteration, a residual flow field is estimated pair-wisely between two frames, and then a bundle adjustment step is applied on all residual flow fields anchored at a keyframe to update the current camera poses and depth. The experiments demonstrate the superiority of the proposed method over existing learning based methods and conventional methods on various datasets.

main_review: Strength:
1. Interlacing between 2D residual flow and 3D camera poses and depth is a nice design to tackle the SLAM problem.
2. The experiments are comprehensive and adequate. The results on various datasets demonstrate the generalization ability of the residual flow estimation.

Weakness:

In general, there is no critical weakness except for some issues that need to be further clarified:
1. The paper does not mention the training data source. It describes how the data is constructed in sec.3.3, but what are the datasets that are used to produce the training data?  Are they from the same dataset as evaluation or they are completely different such as a synthetic dataset? This issue matters to decide how significant the generalization is.
2. The method mentioned that there is a weight predicted in line-177. This weight prediction is important because the method interlacing between residual flow field and bundle adjustment, which is sensitive to outliers such as the ones caused by dynamic objects. However, there is no visualization or evaluation about this weighting in the paper. This issue also depends on the training data that if the training data are all static scenes how it can weigh properly on sequences with dynamic objects? So it would be good to have a clear discussion and analysis in the paper.
limitations_and_societal_impact: The computation limitation has been discussed in line-322. However, the limitation about outlier handling is not mentioned. 
Conventionally, we can do outlier rejection after correspondences have been estimated and then recover camera poses and depth from bundle adjustment. However, in this paper, the flow estimation and bundle adjustment are interlaced. At each iteration, some outlier points such as dynamic objects will always be projected incorrectly and further affect residual flow estimation and bundle adjustment. Unless the weight prediction has been demonstrated to be sufficiently powerful, otherwise it will also be a limitation for this method.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper does dense Structure from Motion using a data-driven dense Bundle Adjustment layer, that is optimized using flow fields. Experiments conducted show that this method outperforms the state of the art and is generalizable to new datasets (which is rare for a learning based method).
main_review: This paper builds on the work done in the RAFT [46] paper, that generates dense optic-flow-fields using an iterative optimization procedure. A 4D dense, multi-scale correlation volume (that is a representation of regions of similarity) is built on pairs of images and the flow-field (and its associated uncertainty) between the pair of images is built up iteratively from this, using a recurrent (GRU) unit. This is like a data-driven, learnt optimization process, that iteratively approaches the correct optic flow given lots of video sequences. However, unlike the RAFT paper, this is not learnt using ground truth flow. In this work, because the structure and motion in the scene needs to be estimated, this is trained using ground truth depth and poses [this is not entirely clear and should be explained better that the ground truth depth and pose are required during training - if I understand it right]. The iterative updates to the optic flow from the GRU unit is in turn used by a Dense Bundle Adjustment (DBA) layer to iteratively arrive at a dense depth map and pose estimate for a pair of images. DBA uses the cost function over a sequence of frames, with flow error (between what is predicted from the GRU and what is obtained from the current estimates of depth and pose) that is optimized using a Gauss-Newton algorithm. It is an important distinction from prior work in this area like BA-net, that used photometric error for optimization and this work uses the flow error. This presumably leads to better robustness to lighting changes and camera blur, etc. The entire network is trained end-to-end using a combination of pose loss and flow loss. 

Experiments conducted indicate the ability of the system to surpass SoTa. A network trained on the Tartan Air dataset generalizes to the EuRoC and TUM-RGBD datasets, and improves on the ATE on those datasets by as much as 83%. 
This is also achieved in real-time or near-real-time run-time on desktop hardware.

This paper is well written and is an impressive result and a significant contribution to the field. 
limitations_and_societal_impact: Some minor grammatical errors that need correction:
126: they
143: show
176: 'convoluation'
227: an
231: 'activly begin'

Around line 123, it is mentioned that the frame graph is built using frame co-visibility, but it is not immediately clear as to how this co-visibility is calculated. It is mentioned later in the paper (207) that this is done by thresholding optic flow. This also needs to be mentioned here (around line 123). It is also not very clear how optic flow is used while training - when presumably the network's own optic flow capabilities will be non-existent (at the beginning of training). Is a pre-trained optic flow network being used for this?

Supervision (209): mentions that a combination of pose loss and flow loss is used and that ground truth poses and depth is used. Does this mean that this network cannot be trained using monocular sequences? Or would you use self-supervised mono-depth techniques to get the depth?
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents R-SLAM, a learning-based SLAM method that shows state-of-the-art camera pose estimation performance on several benchmarks. In particular, R-SLAM updates the camera poses and dense frame depth in an iterative manner by using a recurrent update operator built upon RAFT and differentiable bundle adjustment. To make a complete SLAM system, R-SLAM imposes a covisibility graph and conducts both local and global optimization. The experiments show that R-SLAM outperforms the previous classical and learning-based approaches by a large margin, even on the datasets on which it is not trained.
main_review: ## Strength
1. The proposed SLAM method, R-SLAM, has shown the state-of-the-art camera pose estimation results on several established benchmarks, including TantanAir, EuRoC, TUM RGB-D, and ETH3D. Also, R-SLAM shows in general quite large improvements over the prior works, e.g., by ~60% on TantanAir, ~40% on EuRoC, ~80% on TUM RGB-D, and ~35% on ETH3D.
2. The state-of-the-art results demonstrated are obtained with the model trained on TantanAir only, thus showing very strong generalization capability which is often the major limitation of learning-based SLAM approaches.
3. R-SLAM takes advantage of both classical optimization and deep learning -- it can be trained end-to-end while keeping the bundle adjustment (BA) with Gauss-Newton in the loop. Besides, R-SLAM conducts the joint optimization of pose and depth which has been shown to be one of the keys to accurate estimations.
4. Besides the high accuracy, R-SLAM also shows improved robustness with a high success rate on several datasets.

## Weaknesses
1. In general, the novelty of the paper is not substantial. It builds upon the established systems and methodologies -- a hybrid SLAM design that combines both the classical and learning worlds, differentiable BA / Gauss-Newton layer, covisibility graph, and the recurrent update operator. Overall it feels like a system as DeepV2D + RAFT, and no fundamental theoretical or architectural innovations are proposed. Above being said, I believe combining the previous works and making the right choices that lead to notable improvements is still quite valuable and inspiring to the community.
2. R-SLAM is trained on TatanAir while most of the compared methods (DeepFactors, DeepV2D, D3VO, DeepTAM) are not. Although some of the aforementioned methods are trained with the train/test split on the same data while R-SLAM shows mostly the cross-dataset generalization, it still poses a certain level of unfairness on the comparisons since, in general, the TantanAir dataset contains more images and more diversities. As a suggestion, it would be better to show a method trained and evaluated using exactly the same dataset as R-SLAM and report the comparison, for which DeepV2D could be a good baseline.
3. It is unclear how the non-keyframe poses are estimated if they are estimated at all. Since it is a SLAM system instead of an SfM system, R-SLAM should be able to deliver both keyframe and non-keyframe poses for real-world applications. However, from the paper one cannot see how the non-keyframe poses are handled. In fact, this infers other issues on the evaluation -- R-SLAM is evaluated with the keyframe pose without non-keyframe poses, but is it also the case for the other compared methods?; are the poses used in the evaluations the outputs of the global BA? If so, how good are the poses from local bundle adjustment since some compared methods are Visual Odometry systems without any global BA, e.g, DeepV2D, D3VO, DSO, etc?
4. The details of the full dense bundle adjustment are missing. R-SLAM performs joint dense BA on pose and depth which is quite expensive, especially when doing the global BA which optimizes tens or even hundreds of frame depth and poses. Is there a customized GPU-based optimizer used?
5. It would be better if more qualitative results can be shown, e.g., camera trajectories comparison among the methods and the estimated dense depth maps.
6. The writing of the paper can still be improved. There is a number of typos in the current manuscript. I list some of them here but there should be more: 
- L10: "focused" -> "focuses"
- L13: "approach" -> "approached"
- L15: "are have" -> "have"
- L20: "But despite" -> "Despite" -
- L57: "solves computes" -> "computes" 
- L81: "is that features" -> remove 
- L115: "operators" -> "operates"
- Eq. (5): $\triangle w$ -> $\triangle d$?
7. The name "R-SLAM" already exists in [1] and [2]. In order to avoid confusion in the future, I would suggest the authors proposing a new name of the system.

## Overall Comment
I recommend accepting the paper due to the impressive results and the reasonable design choices made by the authors. But I still would like to see the authors' responses to the points listed in the Weaknesses section above.

----------
References:
1. Balcılar, Muhammet, et al. "R-SLAM: Resilient localization and mapping in challenging environments." Robotics and Autonomous Systems 87 (2017): 66-80.
2. Mei, Christopher, et al. “RSLAM: A system for large-scale mapping in constant-time using stereo. International journal of computer vision 94 (2), 198-214, 2011
limitations_and_societal_impact: The authors have addressed memory consumption as one of the limitations of the method.
review_title: null

Meta-review: The paper received strong reviews and I am happy to recommend it for acceptance. I encourage the authors to make the improvements that they indicated in their rebuttal in the final version. 
Decision: Accept (Oral)