Planning Paths through Occlusions in Urban Environments
Authors: Yutao Han
Youya Xia
Guo-Jun Qi
Mark Campbell
Venue: CoRL 2022 Oral (2022)
Keywords: Navigation
Occluded Environments
Semantic Scene Understanding

Abstract: This paper presents a novel framework for planning in unknown and occluded urban spaces. We specifically focus on turns and intersections where occlusions significantly impact navigability. Our approach uses an inpainting model to fill in a sparse, occluded, semantic lidar point cloud and plans dynamically feasible paths for a vehicle to traverse through the open and inpainted spaces. We demonstrate our approach using a car’s lidar data with real-time occlusions, and show that by inpainting occluded areas, we can plan longer paths, with more turn
options compared to without inpainting; in addition, our approach more closely follows paths derived from a planner with no occlusions (called the ground truth) compared to other state of the art approaches.


Reviews:
[{"review_id": "uvKl0kbSFIp", "reviewer": "Reviewer_apxG", "summary": "In this paper, the authors propose a framework for motion planning under occlusion when driving through dense urban environments. The framework project LiDAR scans onto a BEV (bird eye view) map and fills in unknown regions of the map using a trained inpainting netowrk. The inpainting network is similar to Pix2pix HD, but the authors propose additional training losses to address issues with lost semantic labels from the original map, and missing small details. The framework then uses a hybrid A* planner to plan dynamically feasible paths through the inpainted map", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper addresses an important real world problem and evaluates the effectiveness of the proposed algorithm using the real world KITTI dataset\n- The paper provides statistical results and an example scenario where the proposed framework clearly outperforms previous methods. However, I believe more analysis is needed (see below).\n\nWeaknesses\n1. In the introduction, the authors state that considering unknown spaces as free “increases the chances of varied maneuvering and collisions”. Safety should be considered one of the main motivations of the work. As such, I would like to see experimental results to include metrics that captures this, such as collision rate.\n2. Similarly, the OL baseline seemed to consider unknown spaces as restricted. A comparison should also be done where unknown spaces are considered as free. This baseline will help motivate the work more.\n3. The figures show clear improvement for a clear intersection. However, additional figures should be included to show predictions and planning results in congested intersections, which seem to be a more important use case.\n4. The authors claim that the two additional losses improved the framework's performance. Statistical ablation results should be provided to determine exactly which loss contributes the most to performance of the planner.\n5. This work seems to consider obstacles in the environment as static. In reality, other cars or pedestrians are not static. How does this framework perform in such scenarios? Is this a limitation of the algorithm?\n6. One contribution mentioned in the introduction is a dataset for training occlusion aware models. However, the dataset used in the paper does not seem to be a new dataset, but simply a post processing of an existing dataset. (This is a weak criticism, and may be due to my misunderstanding) ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe that this work addresses an important issue with autonomous driving. The authors provide some results showing scenarios where the algorithm outperform previous methods. However, I do not believe there is enough experimental results that justify that this algorithm is safer than previous methods or will perform better in more important scenarios such as when congestion is occluded. As such, I tentatively recommend this paper to be rejected unless the authors can provide results that addresses the weaknesses stated above.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses above.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "p2nNKL1HVZ", "reviewer": "Reviewer_7Fav", "summary": "This paper presents a new path-planning framework for autonomous vehicle where the visibility is limited to the lidar sensor data. The authors provide a planner based on the hybrid A* algorithm, which plans kinematically feasible paths on 2D grids, with a neural network capable of adding new parts not the grid map based on statistical correlations. \n\nThe noticeable contribution with respect to previous work is to be able to do modify the map without requiring labeling of the pixels that need to be inpainted.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In the recent years, much of machine learning breakthroughs has been in the field of generative models and unsupervised learning. This work leverages state of the art machine learning for dealing with occlusions from sensors. Instead of using a traditional filtering approach the authors purely resort to treating the filtering problem in output space by inpating the grid map’s pixels using a generative model. They make use of computer vision state of the art models to perform this inpainting.\n\nOne of the strength of the paper is to provide dedicated training losses to the problem of inpainting for occlusions. Particularly they introduce reconstruction losses that take care of the zero pixels in the generated images explicitly and a loss that deals with local patches. These modifications quantitatively ameliorate the prediction.\n\nThe paper compares to state of the art inpainting prediction network and is shows to outperform these methods.\nThe authors outline limitations especially concerning the fact that the inpainting behavior is limited to the data and that it relies on exact localization of the agent. \n\nThese make sense and while it would be nice to see extension of the work with other filtering mechanisms to handle uncertainty and data transfer approaches to deal with the data distribution.  The paper seems to adequately represent the state of the art.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a very sound contribution, however potentially incremental. It takes place in a very interesting research avenue, where data driven approaches clearly make sense.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "None.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "IkLdoAF10f9", "reviewer": "Reviewer_hna9", "summary": "This paper presents a new method for predicting occluded spaces.  The proposed inpainting model is then integrated with existing planning methods for navigation in occluded intersections and turns. The benefit of this approach compared to traditional methods that ignore map prediction is that it can design paths that are closer to the paths that the robot would have designed if it had access to the ground truth map. Additionally, the authors provide a dataset for training models of occluded turns and intersections for filling in unknown spaces. The proposed inpainting method employs the Pix2Pix network with a modified loss function while planning is achieved an existing planner, called hybrid A*.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) A dataset for training models of occluded turns and intersections is proposed that can be quite useful for researchers in this area.\n\n2) The proposed method performs better than methods that do not predict the unseen/occluded part of the environment in terms of how close the path is to the path that have been designed if the ground truth map was available.\n\nLimitations:\n1) As the authors admit, a limitation of the proposed method is that it assumes that during online deployment, the data come from the training data distribution. This, then, raises the question whether the proposed method can be really applied to unknown environments or not as it (since training data about the environment is available and at test time the system is applied in the same training environment). The authors should elaborate more on what \"unknown environment\" means.\n\n2) Following up on the previous comment, the benefit/motivation of the proposed method should discussed in more details. Specifically, the above limitation implies that the proposed method can be applied only to known environments for which training data exists. If that's the case, why would an inpainting method be needed? Additionally, the authors should demonstrate more clearly the benefit of the their approach in planning. For instance, how does the (average) re-planning frequency of the proposed method compare to methods that do not predict the structure of occluded environments? In fact, to this reviewer's understanding, high-re-planning frequency is the main limitation of existing planning methods in unknown environments. Does the proposed method alleviate this? Also, it is worth noting that there exists a long list of works on navigating unknown environments without requiring (re)-planning. For instance  potential fields-based controllers for various tasks in unexplored semantic worlds have been proposed such as the following ones:\n\nArslan, O., & Koditschek, D. E. (2016, December). Sensor-based reactive navigation in convex sphere worlds. In The 12th International Workshop on Algorithmic Foundations of Robotics (WAFR).\n\nVasilopoulos, Vasileios, et al. \"Reactive planning for mobile manipulation tasks in unexplored semantic environments.\" 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.\n\nWhat is the benefit of the proposed method against such works?\n\n3) The literature review can be improved as there are several works focusing on predicting the structure of unseen environments; see e.g., the following papers and the references therein:\nGeorgakis, G., et al (2022). Uncertainty-driven planner for exploration and navigation. IEEE Conference on Robotics and Automation (ICRA), 2022\n\nM. Narasimhan et al “Seeing the un-scene: Learning amodal semantic maps for room navigation,” in European Conference\non Computer Vision. Springer, 2020, pp. 513–529\n\n4) Also, obviously, the proposed method cannot have 100% accuracy. It would be interesting to provide cases where e.g., the proposed method predicted an intersection which did not exist. How is planning affected by such wrong predictions?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper addresses a very interesting and important problem while the simulation studies show that it can outperform existing methods. This reviewer's major concern is about the contribution of the proposed algorithm as well as its practical importance compared to existing methods. As mentioned earlier, there are several methods that focus on predicting the unseen part of unknown environments. Also, the authors should elaborate more on the benefit of the proposed method with respect to planning and provide more detailed comparisons (see limitations 1-2).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1) The authors should provide a more thorough literature review and discussing how this method compares against related works.\n2) The benefit of the proposed method against planning methods that consider the unknown space as free should be better explained. For instance, the replanning frequency should be reported.\n3) If space permits, this reviewer would encourage the authors to include figures where the proposed method does not predict an accurate map along with the designed paths."}}, {"review_id": "C0trGJHseeR", "reviewer": "Reviewer_dgc4", "summary": "In this paper the authors leverage an existing image generation neural network to fill in (or in-paint) missing and occluded drivable areas of the ground vehicle’s sensor map. The input is a bird’s eye view translation of a semantic LIDAR sensor. The neural net is trained with a custom loss function intended to increase in-painting area and preserve small features. The resulting image prediction is used as the basis for an occupancy map.\n\nThe authors use the Hybrid-A∗ algorithm on the prediction map to plan a trajectory that closely matches what would be generated if no occlusions were present.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors illustrate nicely the ability of the trained system/GAN to approximate a local map from only LIDAR data\n\n- The results when compared to Pix2PixHD and DeepFillV2 are impressive with clear improvements.\n\nWeaknesses:\n- There is no comparison of results with the paper that most closely resembles their work (i.e., ref [5] in the paper). Instead, they compare only against generic in-painting algorithms (Pix2Pix, Pix2PixHD, and DeepFillV2) then plan using their motion planner. \n\n- In the introduction, the authors comment that the two current approaches to planning in an occluded environment are to either plan as if space is free, or plan only in the visible map. They comment that if planning as if space is free, then they have to consider what may be in the space beyond the occlusion, e.g., pedestrians. However, in this paper, the probable free space is constructed and then used for planning – without regard to what may be occupying that space.\n\n- There is no discussion about the cases in which this method fails and is incorrect in its prediction of free space. This is very important when considering safety.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is a nice demonstration of adapting an existing in-painting neural net to predict the static shape of occluded space. The innovations seem limited to customizing the loss function and using a dynamics-based motion planner. While the authors are aware of a very similar paper that uses similar technology, they don’t adequately develop their improvements over this existing work. Finally, while they raise the safety issue of planning as if space is free, their algorithm subsequently plans trajectories as if there are no dynamic agents (e.g., pedestrians) in the occlusions.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- It is not entirely clear to this reviewer what the main improvements are over [5] as comparisons are made against generic in-painting algorithms. The main difference as I understand is that they use a different GAN and that [5] requires an explicit labeling (i.e., a mask to remove the areas that do not need to be in-painted), and the use of Hybrid-A* instead of A*.  More discussion is needed on why this explicit labelling is such a significant drawback. \n\n- The introduction raises the safety issue of planning as if space is free, but then does so. It is not clear how this approach is different/better than having a coarse map.\n\n- The paper could use a stronger motivation for when no coarse map of the environment would be availablpe a priori.  \n\n- The comparison to Pix2Pix does not seem necessary as Pix2PixHD is an improvement on that.\n\n******************************************************\nRevised review:  Based on the reviewer responses I am updating my rating to weak accept and removing criticisms about the lack of consideration for dynamic obstacles\n*******************************************************"}}]

Author rebuttal:

Decision: Accept (oral)