Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper describes a Deep Variational Bayes Filter (DBVF) for Deep-Learning based SLAM in 3D environments. It builds upon similar work for 2D environments in [Mirchev et. al. 19], and learns a full 3D RGBD occupancy map and a sequence of 6 DoF poses (localization) using raw stereoscopic camera data. Differentiable ray-casting and an attention model is described to access the learnt global map to give a local map and an expected observation - using an emission model from the current pose and local map. A transition model describing the evolution of the dynamics of the agent is also learnt. A variational approximation of the actual posterior (of the sequence of poses and the map, given the sequence of observations) is learnt by optimizing the standard ELBO equation from Variational Bayes. Such deep generative models, once learnt (in an unsupervised way) for an environment, allows one to hallucinate a sequence of poses and observations, given the learnt map and control inputs. This allows downstream robotic control tasks like environment exploration and path planning to be integrated into the model. Experiments on a simulated dataset with a flying drone in a subway and living room environments demonstrate good SLAM performance (that approach traditional methods): bird's eye view projections of the 6 DoF poses and the emitted maps closely match the ground truth poses and the occupancy grid. This is a well-written paper that represents a good step up from [Mirchev et. al. 19] to formulate a DVBF with realistic RGBD data streams. The authors mention that the computational times for this method is still far from conventional SLAM techniques - an actual quantification of the time taken during inference would be useful.
review_title: A good paper - accept.

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
## Summary

The paper proposes a framework built on DVBF-LM, extended to dense 3D mapping. Overall I find the work somewhat incremental over DVBF-LM and that the methodology lacks clarity. 

## Strengths

 - The work builds on a fundamentally new and interesting line of generative variational approaches to SLAM
 - It also builds on recent work in differentiable SLAM systems
 - It shows results on datasets taken from an actual quadrotor.
 - Results seem promising (but comparisons to some other related approaches might be missing)
 

## Weaknesses

 - I found the methodology extremely difficult to follow. There are many variables used, and some choices are simply not explained or not clear. This becomes very laborious. One such example is that the variable $o$ is used to represent occupancy as well as the agent location in space. In all these variables, I missed a definition for $q_\phi$. Part of this is also that non-standard terms and notation are used. As far as I know, "map charts" is not a common term. Neither is emission model (should be measurement model). Although I do see that these notations are borrowed from DVBF-LM.

 - I feel there are some related works that are missing from the discussion. The approach in [1] formulates the fully differentiable dense SLAM problem (including ray casting) on real world datasets, [2] is a recent novel 3D rendering approach and also does differentiable ray casting,  [3] demonstrates an end to end approach to learning measurement likelihood models with an RL active localization framework, and [4] which is also uses a clever combination of deep learning and multi-view geometry to produce dense 3D maps. There are others. In general, differentiable ray casting is not new, and the authors should make a more concerted effort to cite this work and place their approach within this context.  

[1] ∇ SLAM: Dense SLAM meets Automatic Differentiation
KM Jatavallabhula, G Iyer, L Paull
2020 IEEE International Conference on Robotics and Automation (ICRA), 2130-2137

[2] Neural Reflectance Fields for Appearance Acquisition
S Bi, Z Xu, P Srinivasan, B Mildenhall, K Sunkavalli, M Hašan, ...
arXiv preprint arXiv:2008.03824

[3] Active Neural Localization
DS Chaplot, E Parisotto, R Salakhutdinov
6th International Conference on Learning Representations (ICLR-18)

[4] Towards the Probabilistic Fusion of Learned Priors into Standard Pipelines for 3D Reconstruction
T Laidlow, J Czarnowski, A Nicastro, R Clark, S Leutenegger
2020 IEEE International Conference on Robotics and Automation (ICRA), 7373-7379

[5] Neural scene representation and rendering
SMA Eslami, DJ Rezende, F Besse, F Viola, AS Morcos, M Garnelo, ...
Science 360 (6394), 1204-1210

 - It would nice to know in the related work section what exactly it is about previous methods that makes them not applicable to "real-world 3D modelling". In particular, the very related DVBF-LM which method builds upon. 

 - The parts of the "Method" section which are taken from DVBF-LM should be moved to a section called "Background." That would enable the reader to more clearly differentiate what is new here. 

## Other Comments/Questions

 - You state that " $M^col$ represents the parameters of a feed-forward neural network $f_{M^col}: R3 \mapsto [0, 255]^3$. The network assigns an RGB colour value to each point in space." What colours are assigned to free space? What about viewpoint dependent illumination?
review_title: ICLR 2021

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a novel learning-based visual-inertial odometry algorithm. The algorithm simultaneously reconstructs the world map as well as the states of the agent from the stereo RGBD sensors.  The world is modeled as an occupancy grid with color. A graphical model with attention mechanism and ray casting is used to model how the world and the agent state renders the RGBD sensor data. ELBO is used to optimize the model. The technical details look sound to me.

The learning part of the paper is to use neural networks to model the dynamics: the distribution of the next agent state given the current state and the IMU is modeled as a normal distribution whose mean and standard deviation are determined via a neural network that takes the current state and the IMU as the input. The neural network is trained on pre-recorded agent states and IMU data captured in the MOCAP. Note that the neural network builds upon the Euler integration based IMU model; it models the residual between the true agent next state and the prediction using the Euler-integration based IMU model. 

The paper performs extensive experiments on a dataset that includes agents of different velocities and different types of trajectories. Qualitatively the trajectories generated by the proposed method look great in the figures. Quantitatively, the average localization error is around 0.2% of the trajectory length, demonstrating the practicality of the method. 

However, compared with the SOTA methods, the proposed method is slower and at the same time has higher localization errors. In addition, the learning-based dynamics model does not outperform the Euler integration based IMU model, despite the former builds upon the latter. The paper also reports two outlier data points in the experiments where the learning-based dynamics model leads to significantly higher localization errors. In the appendix, the authors analyze these two data points and identify the cause: the landing of the agents in these two trials fail and lead the model to learn ungeneralizable dynamics.

The presentation of the paper is great. The technical parts of the paper are written concisely and precisely with clear motivation and intuition. I find the paper easy to understand. 

To summarize:

Pros:
1. Great presentation
2. Novel formulation and learning-based dynamics model
3. Extensive experiments

Cons:
1. The proposed method is behind the state-of-the-art in both speed and localization accuracy
2. The learning-based dynamics model proposed by the model does not outperform the closed-form Euler integration based one
review_title: A novel learning based SLAM algorithm, but slower and less precise than SOTA methods 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

This paper addresses the problem of dense RGB-D SLAM. The key idea is to formulate the problem as a deep state-space model and infer the state of the latent variables (i.e. pose and geometry) using variational inference. The experiments demonstrate that the method performs well in a challenging quadcopter dataset. However, the advantages of the approach are not demonstrated. 

Strengths
- The proposed approach is a very elegant and principled formulation of the dense SLAM problem. The probabilistic graphical model considers the observations, dynamics and latent states of the agent (i.e. the pose and dense map). As far as I am aware, not many deep slam approaches consider all these factors in the estimation process. The proposed model is therefore a nice unification of neural networks, dynamics and multi-view geometry.

- The method has a number of interesting capabilities such as the ability to predict the future trajectory of the camera. The demonstration of the usefulness of the inferred map in downstream tasks such as navigation is also very interesting.

Weaknesses
- The related work is severely lacking. Specifically, the paper seems to marginalise most of the recent methods that try to combine deep learning with dense SLAM. The authors summarize the whole collection of such approaches with a single sentence: “depth or semantic representations with existing SLAM methods has also become a prevalent direction of research”. This isn’t a fair representation of existing works and doesn’t really do much to position the authors’ novel contribution with respect to the existing literature. The authors need to discuss and contrast their approach to works such as BA-Net, DeepFactors , Neural RGB->D, DeepSFM etc. Currently, the related work is too focussed on traditional SLAM approaches (like VinsMono) and 2D/2.5D methods like VAST/DVBF-LM.

- The proposed method requires RGB-D as input which is obtained from SGM in the paper. How sensitive is the approach to noise in the depth input, and can the approach work with just RGB input? The reason I consider this a weakness is that many existing RGB-D SLAM approaches (even traditional ones like ORB-SLAM2) have very high accuracy and in some sense can be considered a “solved problem”. This is not really the case for dense monocular SLAM which is still a very challenging problem.

- Furthermore, the advantages of the proposed approach are not described in the paper. In the evaluation the improvement over ORB-SLAM2 in terms of accuracy doesn’t appear to be very significant. Is it simply an improvement in accuracy or are there other advantages as well? 

- The experiments and comparisons are also rather limited. The authors have only evaluated on a single quadcopter dataset which doesn’t really show the generalizability of the trained model. ORB-SLAM2 and VINS-MONO don’t have any trained components and are therefore completely general (apart from the need for tuning some settings). How does the proposed approach perform when trained on Blackbird and evaluated on EuRoC, for example? 

- The runtime of SLAM approaches is quite important, especially for application on platforms such as quadcopters. It would be good if the authors could include some indication of the runtime and whether the inference can be done in realtime.

Minor points

It would be good to bold the best performing methods in the table as this makes it easier to see the relative performance of the methods.


review_title: Good formulation of dense RGB-D SLAM as a deep state-space model, but what are the advantages?

Meta-review: The paper proposes a method for SLAM like dense 3D mapping (colored occupancy grid) based on differentiable rendering with a possibility to provide a probabilistic generative predictive distribution, evaluated on UAVs.

Initially this paper has a wide spread of reviews, with ratings between 4 and 9. Reviewers appreciated the elegant and principled formulation and the interest of the predictive distribution. On the downside, several issues were raised on the incremental nature wrt to DVBF-LM; presentation and writing being very dense and difficult to follow; positioning wrt to prior art; performance with respect to known visual SLAM SOTA baselines; limited evaluations.

The authors provided responses to many of this questions and also updates to the paper, which convinced several reviewers, who unanimously recommended acceptance after discussion.

The AC concurs.
Decision: Accept (Poster)