Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a new path-planning framework for autonomous vehicle where the visibility is limited to the lidar sensor data. The authors provide a planner based on the hybrid A* algorithm, which plans kinematically feasible paths on 2D grids, with a neural network capable of adding new parts not the grid map based on statistical correlations. 

The noticeable contribution with respect to previous work is to be able to do modify the map without requiring labeling of the pixels that need to be inpainted.
strengths_and_weaknesses: In the recent years, much of machine learning breakthroughs has been in the field of generative models and unsupervised learning. This work leverages state of the art machine learning for dealing with occlusions from sensors. Instead of using a traditional filtering approach the authors purely resort to treating the filtering problem in output space by inpating the grid map’s pixels using a generative model. They make use of computer vision state of the art models to perform this inpainting.

One of the strength of the paper is to provide dedicated training losses to the problem of inpainting for occlusions. Particularly they introduce reconstruction losses that take care of the zero pixels in the generated images explicitly and a loss that deals with local patches. These modifications quantitatively ameliorate the prediction.

The paper compares to state of the art inpainting prediction network and is shows to outperform these methods.
The authors outline limitations especially concerning the fact that the inpainting behavior is limited to the data and that it relies on exact localization of the agent. 

These make sense and while it would be nice to see extension of the work with other filtering mechanisms to handle uncertainty and data transfer approaches to deal with the data distribution.  The paper seems to adequately represent the state of the art.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper presents a very sound contribution, however potentially incremental. It takes place in a very interesting research avenue, where data driven approaches clearly make sense.
issues: None.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper, the authors propose a framework for motion planning under occlusion when driving through dense urban environments. The framework project LiDAR scans onto a BEV (bird eye view) map and fills in unknown regions of the map using a trained inpainting netowrk. The inpainting network is similar to Pix2pix HD, but the authors propose additional training losses to address issues with lost semantic labels from the original map, and missing small details. The framework then uses a hybrid A* planner to plan dynamically feasible paths through the inpainted map
strengths_and_weaknesses: Strengths
- The paper addresses an important real world problem and evaluates the effectiveness of the proposed algorithm using the real world KITTI dataset
- The paper provides statistical results and an example scenario where the proposed framework clearly outperforms previous methods. However, I believe more analysis is needed (see below).

Weaknesses
1. In the introduction, the authors state that considering unknown spaces as free “increases the chances of varied maneuvering and collisions”. Safety should be considered one of the main motivations of the work. As such, I would like to see experimental results to include metrics that captures this, such as collision rate.
2. Similarly, the OL baseline seemed to consider unknown spaces as restricted. A comparison should also be done where unknown spaces are considered as free. This baseline will help motivate the work more.
3. The figures show clear improvement for a clear intersection. However, additional figures should be included to show predictions and planning results in congested intersections, which seem to be a more important use case.
4. The authors claim that the two additional losses improved the framework's performance. Statistical ablation results should be provided to determine exactly which loss contributes the most to performance of the planner.
5. This work seems to consider obstacles in the environment as static. In reality, other cars or pedestrians are not static. How does this framework perform in such scenarios? Is this a limitation of the algorithm?
6. One contribution mentioned in the introduction is a dataset for training occlusion aware models. However, the dataset used in the paper does not seem to be a new dataset, but simply a post processing of an existing dataset. (This is a weak criticism, and may be due to my misunderstanding) 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I believe that this work addresses an important issue with autonomous driving. The authors provide some results showing scenarios where the algorithm outperform previous methods. However, I do not believe there is enough experimental results that justify that this algorithm is safer than previous methods or will perform better in more important scenarios such as when congestion is occluded. As such, I tentatively recommend this paper to be rejected unless the authors can provide results that addresses the weaknesses stated above.
issues: See weaknesses above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper the authors leverage an existing image generation neural network to fill in (or in-paint) missing and occluded drivable areas of the ground vehicle’s sensor map. The input is a bird’s eye view translation of a semantic LIDAR sensor. The neural net is trained with a custom loss function intended to increase in-painting area and preserve small features. The resulting image prediction is used as the basis for an occupancy map.

The authors use the Hybrid-A∗ algorithm on the prediction map to plan a trajectory that closely matches what would be generated if no occlusions were present.

strengths_and_weaknesses: Strengths:
- The authors illustrate nicely the ability of the trained system/GAN to approximate a local map from only LIDAR data

- The results when compared to Pix2PixHD and DeepFillV2 are impressive with clear improvements.

Weaknesses:
- There is no comparison of results with the paper that most closely resembles their work (i.e., ref [5] in the paper). Instead, they compare only against generic in-painting algorithms (Pix2Pix, Pix2PixHD, and DeepFillV2) then plan using their motion planner. 

- In the introduction, the authors comment that the two current approaches to planning in an occluded environment are to either plan as if space is free, or plan only in the visible map. They comment that if planning as if space is free, then they have to consider what may be in the space beyond the occlusion, e.g., pedestrians. However, in this paper, the probable free space is constructed and then used for planning – without regard to what may be occupying that space.

- There is no discussion about the cases in which this method fails and is incorrect in its prediction of free space. This is very important when considering safety.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper is a nice demonstration of adapting an existing in-painting neural net to predict the static shape of occluded space. The innovations seem limited to customizing the loss function and using a dynamics-based motion planner. While the authors are aware of a very similar paper that uses similar technology, they don’t adequately develop their improvements over this existing work. Finally, while they raise the safety issue of planning as if space is free, their algorithm subsequently plans trajectories as if there are no dynamic agents (e.g., pedestrians) in the occlusions.
issues: - It is not entirely clear to this reviewer what the main improvements are over [5] as comparisons are made against generic in-painting algorithms. The main difference as I understand is that they use a different GAN and that [5] requires an explicit labeling (i.e., a mask to remove the areas that do not need to be in-painted), and the use of Hybrid-A* instead of A*.  More discussion is needed on why this explicit labelling is such a significant drawback. 

- The introduction raises the safety issue of planning as if space is free, but then does so. It is not clear how this approach is different/better than having a coarse map.

- The paper could use a stronger motivation for when no coarse map of the environment would be availablpe a priori.  

- The comparison to Pix2Pix does not seem necessary as Pix2PixHD is an improvement on that.

******************************************************
Revised review:  Based on the reviewer responses I am updating my rating to weak accept and removing criticisms about the lack of consideration for dynamic obstacles
*******************************************************
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a new method for predicting occluded spaces.  The proposed inpainting model is then integrated with existing planning methods for navigation in occluded intersections and turns. The benefit of this approach compared to traditional methods that ignore map prediction is that it can design paths that are closer to the paths that the robot would have designed if it had access to the ground truth map. Additionally, the authors provide a dataset for training models of occluded turns and intersections for filling in unknown spaces. The proposed inpainting method employs the Pix2Pix network with a modified loss function while planning is achieved an existing planner, called hybrid A*.
strengths_and_weaknesses: Strengths:
1) A dataset for training models of occluded turns and intersections is proposed that can be quite useful for researchers in this area.

2) The proposed method performs better than methods that do not predict the unseen/occluded part of the environment in terms of how close the path is to the path that have been designed if the ground truth map was available.

Limitations:
1) As the authors admit, a limitation of the proposed method is that it assumes that during online deployment, the data come from the training data distribution. This, then, raises the question whether the proposed method can be really applied to unknown environments or not as it (since training data about the environment is available and at test time the system is applied in the same training environment). The authors should elaborate more on what "unknown environment" means.

2) Following up on the previous comment, the benefit/motivation of the proposed method should discussed in more details. Specifically, the above limitation implies that the proposed method can be applied only to known environments for which training data exists. If that's the case, why would an inpainting method be needed? Additionally, the authors should demonstrate more clearly the benefit of the their approach in planning. For instance, how does the (average) re-planning frequency of the proposed method compare to methods that do not predict the structure of occluded environments? In fact, to this reviewer's understanding, high-re-planning frequency is the main limitation of existing planning methods in unknown environments. Does the proposed method alleviate this? Also, it is worth noting that there exists a long list of works on navigating unknown environments without requiring (re)-planning. For instance  potential fields-based controllers for various tasks in unexplored semantic worlds have been proposed such as the following ones:

Arslan, O., & Koditschek, D. E. (2016, December). Sensor-based reactive navigation in convex sphere worlds. In The 12th International Workshop on Algorithmic Foundations of Robotics (WAFR).

Vasilopoulos, Vasileios, et al. "Reactive planning for mobile manipulation tasks in unexplored semantic environments." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.

What is the benefit of the proposed method against such works?

3) The literature review can be improved as there are several works focusing on predicting the structure of unseen environments; see e.g., the following papers and the references therein:
Georgakis, G., et al (2022). Uncertainty-driven planner for exploration and navigation. IEEE Conference on Robotics and Automation (ICRA), 2022

M. Narasimhan et al “Seeing the un-scene: Learning amodal semantic maps for room navigation,” in European Conference
on Computer Vision. Springer, 2020, pp. 513–529

4) Also, obviously, the proposed method cannot have 100% accuracy. It would be interesting to provide cases where e.g., the proposed method predicted an intersection which did not exist. How is planning affected by such wrong predictions?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, the paper addresses a very interesting and important problem while the simulation studies show that it can outperform existing methods. This reviewer's major concern is about the contribution of the proposed algorithm as well as its practical importance compared to existing methods. As mentioned earlier, there are several methods that focus on predicting the unseen part of unknown environments. Also, the authors should elaborate more on the benefit of the proposed method with respect to planning and provide more detailed comparisons (see limitations 1-2).
issues: 1) The authors should provide a more thorough literature review and discussing how this method compares against related works.
2) The benefit of the proposed method against planning methods that consider the unknown space as free should be better explained. For instance, the replanning frequency should be reported.
3) If space permits, this reviewer would encourage the authors to include figures where the proposed method does not predict an accurate map along with the designed paths.
review_title: null

Meta-review: metareview: Summary 
The authors present a framework for motion planning in congested urban areas. The framework project LiDAR scans a BEV map and fills in unknown parts using a trained inpainting network. The inpainting network is comparable to Pix2pix HD. 
The proposed inpainting model is then combined with existing navigation planning methods. This strategy can create paths closer to what the robot would have designed with the ground truth map than standard methods that disregard map prediction.
The authors propose extra training losses to address missing semantic labels and minor details. A hybrid A* planner plans dynamically feasible paths through the inpainted map. 


Strengths 
1. A dataset for training models of occluded turns and junctions is proposed. 
2. The suggested strategy outperforms methods that do not forecast the unseen/occluded component of the environment. 
3. Results are better than Pix2PixHD and DeepFillV2. 
4. The research tests the suggested technique using the real-world KITTI dataset. 
5. The paper offers statistical findings and a situation where the suggested framework outperforms existing methods. More research is needed. 
6. Recent machine learning advancement has been in generative models and unsupervised learning. 
7. This work uses machine learning to handle sensor occlusions. Instead of a standard filtering strategy, the authors inpaint grid map pixels with a generative model. Inpainting uses cutting-edge computer vision models. 
8. The paper provides separate training losses for occlusion inpainting. They introduce reconstruction losses for zero pixels and local patches. This improves prediction quantitatively. 
9. The paper outperforms state-of-the-art inpainting prediction networks. The authors note that inpainting is restricted to data and requires precise agent location.

Update: 
The concerns raised have been clarified and the revised submission substantially updated. 

Weaknesses
1. The authors say that interpreting unknown locations as free "increases maneuvering and collisions." Safety should be a top priority. I'd want to see experimental findings that include collision rate. 
2. OL seems to restrict unknown regions. Unknown spaces should also be compared. 
3. Figures show an improved junction. Additional data should show predictions and planning results in congested intersections, a more important use case. 
4. Two additional losses improved the framework's performance, say the authors. Statistical ablation findings should show which loss contributes more to planner performance. 
5. This work considers environmental obstacles static. Cars and pedestrians aren't static. How does this framework perform? Is this a programming error? 
6. A dataset for training occlusion-aware models was given in the introduction. The paper's dataset appears to be a post-processing of an existing dataset. (This is a weak criticism owing to my misunderstanding.) 
7. No comparison of findings with similar paper (i.e., ref [5] in the paper). Instead of comparing to generic in-painting algorithms, they use their motion planner. 
8. The two current approaches to planning in an occluded environment are to plan as if space is free or to plan just in the viewable map. They note that if they design as if there is no occlusion, they must include pedestrians. In this paper, however, the expected open space is built and then used for planning. 
9. It's not clear when this algorithm anticipates empty space wrongly. Safety requires this. 
10. The authors admit that online deployment presupposes training data is used. The proposed method's applicability to unknown contexts is questioned (since training data about the environment is available and at test time the system is applied in the same training environment). "Unknown environment" needs clarification. 
11. The proposed method's benefit/motivation should be addressed in more detail.

confidence: 5: The area chair is absolutely certain
Decision: Accept (Oral)