Context-Aware Deep Reinforcement Learning for Autonomous Robotic Navigation in Unknown Area
Authors: Jingsong Liang
Zhichen Wang
Yuhong Cao
Jimmy Chiun
Mengqi Zhang
Guillaume Adrien Sartoretti
Venue: CoRL 2023 Poster (2023)
Keywords: deep reinforcement learning
mapless navigation
context-aware decision-making

Abstract: Mapless navigation refers to a challenging task where a mobile robot must rapidly navigate to a predefined destination using its partial knowledge of the environment, which is updated online along the way, instead of a prior map of the environment. Inspired by the recent developments in deep reinforcement learning (DRL), we propose a learning-based framework for mapless navigation, which employs a context-aware policy network to achieve efficient decision-making (i.e., maximize the likelihood of finding the shortest route towards the target destination), especially in complex and large-scale environments. Specifically, our robot learns to form a context of its belief over the entire known area, which it uses to reason about long-term efficiency and sequence show-term movements. Additionally, we propose a graph rarefaction algorithm to enable more efficient decision-making in large-scale applications. We empirically demonstrate that our approach reduces average travel time by up to $61.4\%$ and average planning time by up to $88.2\%$ compared to benchmark planners (D*lite and BIT) on hundreds of test scenarios. We also validate our approach both in high-fidelity Gazebo simulations as well as on hardware, highlighting its promising applicability in the real world without further training/tuning.

Context-Aware Deep Reinforcement Learning for Autonomous Robotic Navigation in Unknown Area
Jingsong Liang" Zhichen Wang" Yuhong Cao Jimmy Chiun
Mengqi Zhang Guillaume Sartoretti
National University of Singapore
(jingsongliang, zhichenwang, caoyuhong, jimmy. chiun S nus edu {mpezmq, gui guillaume sartoretti @nus. maus.edda edu.sg
Abstract:
Mapless navigation refers to challenging task where mobile robot must rapidly navigate to predefined destination using its partial knowledge of the environ- ment, which is updated online along the way, instead of prior map of the environ- ment. Inspired by the recent developments deep reinforcement learning (DRL), we propose learning-based framework for mapless navigation, which employs context-aware policy network achieve efficient decision-making (i.e., maxi- mize the likelihood of finding the shortest route towards the target destination), especially in complex and large-scale environments. Specifically, to form context of its belief over the entire known area, which it uses to reason about long-term efficiency and sequence show-term movements. Additionally, we propose graph rarefaction algorithm to enable more efficient decision decision-making large-scale applications. We empirically demonstrate that our approach reduces average travel time by up to 61.4% and average planning time by up to 88.2% compared to benchmark planners (D*lite and BIT) on hundreds of test scenarios. We also validate our approach both in high-fidelity Gazebo simulations well on hardware, highlighting its promising applicability in the real world without further training/tuming.
robot learns
Keywords: deep reinforcement learning, mapless navigation, context-aware decision-making
Introduction
Autonomous navigation is essential capability for mo- bile robots, and can be broadly divided into local and global planning. Local planning typically focuses on short-term collision avoidance, which provides the robot with reactive kinematic commands to navigate through its nearby surroundings [1. 2]. Global planning, on the other hand, requires the robot consider the broader environ- mental information and provide movement decisions at higher level to determine long-term route towards the target destination [3, 4]. Although many works have stud- ied map-based navigation, where the robot relies on prior information about the environment, in this work, we fo- cus on global planning for mapless navigation, where robot starts navigation the destination in completely unknown/unmapped environment. Throughout the task,
Figure Illustration of navigation through unknown environment. The ground vehicle generates a feasible trajectory toward the destination relying on the sensory inputs and destination.
'These authors contributed equally to this work.
Corresponding authors.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

the robot incrementally constructs/updates its partial belief/map of the environment using onboard sensors, to guide its global path planning. The robot is tasked with reaching the target destination as quickly as possible (in other words, exploring the least amount of the environment).
Although map-based navigation has been relatively well-studied [5, 6, 7J. mapless navigation re- mains open challenge: it requires the robot to explicitly or implicitly predict the potential path to the target destination based on partial knowledge of the environment. For example, conventional approaches often assume the shortest path lies behind the nearest frontier (i.e., the boundary between traversable and unknown areas) to the destination. Such predictions naturally have a significant im- pact on navigation efficiency; however, improving the accuracy of these predictions is non-trivial, especially in complex scenarios with plenty of dead-ends and dense obstacles. We believe existing approaches lack comtexl-awareness when predicting potential paths. Being context-aware means that the robot is capable of adaptively reason about potential global paths according to its belief over the current environment, instead of following fixed rule for all scenarios that may be significantly different. It also allows the robot to make non-myopic decisions that benefit long-term efficiency and maximize the likelihood of reaching the destination as fast as possible.
In this paper, we investigate and propose a context- aware DRL framework for mapless navigation. Specifically, our robot reasons about the context of its belief over the entire known area using an attention- based neural network. During the navigation task, our robot builds and updates its belief (represented as graph), which serves as input of our neural network. The policy network models the inter-dependencies between distinct areas in the agent's 's belief/graph, to finally output the next waypoint for navigation. We further propose a graph rarefaction algorithm to filter out redundant nodes and corresponding edges in this graph for more efficient policy learning in large-scale envi- ronments. Compared to existing learning-basee based approaches [8. 9, 10], our model not only outputs a near-optimal policy sequence movement decisions towards the target destination, but also endows the agent with time-efficient re- re-planning abilities even in complex, large-scale environments. To sess the performance and generalizability of our approach, we compare our model to representative statc-of-ther -orth-rr- baselines in hundreds of simulated maps at three complexity levels, where we high- light improvements up to 61% for average travel time and 88% for average planning time over these baselines. We also validate the effectiveness of our approach in large-scale Gazebo simulations, and deploy our planner on hardware in a real-world scenario. To the best of our knowledge, we are the first work that proposes a DRL-based approach to mapless navigation [11]. Our full code and trained model is available at https: / htt:://gitttt com/ com/narmottaabConteee wwwpennttettttttteeeeeeeeeeeeeee Axware_Mavggation
2 Related Works
These approaches are primarily based on discrete grids or a graph with node priority evaluation, such Dijkstra [12] and A [5]. Given prior map, they are able to search for npar-optimal paths [6]. Dynamic variants of search-based approaches, e.g., D* [13], LPA* * [3], D* lite [4], were proposed to approach dynamic planning in unknown environments. In most cases, these algorithms replan at low frequency, by reusing the planned route from the previous search. However, a robot may have to replan a new feasible route upon realizing that the current one is impractical (c.g., reaching a dead-end) [14]. In this case, such incremental search comes at a high computational cost, especially in complex environments. Furthermore, these algorithms are highly dependent on heuristic function, which results in them being neither cognitive nor robust [15].
Search-based approaches:
Sampling-based approaches: More recently, the rapidly-exploring Random Tree (RRT) [16] fam- ily, which includes original RRT. RRT* [17], and RRT-Connect [18], has been leveraged for path planning due to its low computation costs. Owing to the drawback of random sampling, the gener- ated paths are typically sub-optimal and unstable [19], and are prone to being stuck in local minima, especially in complex environments. To achieve effective sampling, several improvements have been proposed [20, 21, 22, 23). These approaches perform well in map-based navigation [7], but typically suffer from long planning time in the absence of prior information [24], mainly due to extensive sampling in both known and unknown areas.

---

Current Encoded Embeddings
Mask
Policy
Mask
Action
Viewpoint Graph
Refined Viewpoint Graph
Encoder
Decoder
Target Node
Node Features
Encoded Embeddings
Neighboring Encoded Embeddings
Figure 2: Framework of our context-aware DRL policy network. The initial and refined view-
point graphs are built from the agent's partial belief and graph rarefaction, respectively. The encoder first incorporates global information from the current partial map and the destination node through self-attention. This embedded information is then used by the decoder to reason about the depen- dencies between the current node and its neighbors, to finally generate the action policy.
Learning-based approaches: Learning-basse mapless navigation approaches have aroused interest and demonstrated effectiveness recent years [2, 9]. They can be broadly subdivided into local and global planning. Most of the previous learning-based works have focused on local planning through end-to-end models [11], including vision-based, i.e., successor-eattueebeee based [8], LiDAR- based [9, 10, 25], or imitation imitation-learning based approaches [26, 27]. Both [9] and [28] note that global path planner should usually be used for trust-worthy navigation in unknown environments. Furthermore, these local planners are typically trained and validated in simple environments, raising concerns about their generalizability to more complex cases. To the best of our knowledge, we are the first work that proposes a learning-bas -ased global planner for mapless navigation [11].
3 Approach
In this section, we consider mapless navigation sequential decision-making RL problem and de- tail our context-aware policy network, as well as our graph rarefaction algorithm to further improve longer-horizon and larger-scale planning.
3.1 Mapless navigation as RL Problem
We formulate the mapless navigation task as partially observable Markov decision process (POMDP), expressed as a tuple (S, A, T, R. Ω O)7) with the state space S, the action space A, the state transition function T. the reward function R, the observation space Ω the observation function given the true state s' of the environment O (o, 2 a) and the action state a, and the discount fac- tor To promote efficient navigation towards the target destination, the RL objective is aimed find Σ 7d Ir
an optimal policy n² that maximizes the expected discounted reward (aa-s(()oe) The policy π can be considered as a mapping function from og to the next action dz.
or (G,, St), which consists of the viewpoint graph G, and the planning attributes Sc. The robot gets the updated obser- vation in the limited sensor range d, (80 in practice training). The agent first obtains the viewpoint set Vg, which is generated uniformly in the known area D. To construct traversable graph, every node in Vú constructs up to nearest edges with each other, where the edges in E, can only connect between collision-fii nodes, i.c. nodes that are line of sight with each other. The construction of G, (V,, F.)) eliminates the concerns of collision with obstacles. Moreover, the planning attributes S, provide additional information about the observed environment and the target for each node in V,. Inspired by [29], every node has direction vector ö which acts signpost for the target, which consists of unit vector indicating the direction towards the target and the Euclidean distance
Observation Ω: At each decision step our observation is represented

---

[ū] from the node to the target. S; further include an indicator ó; which records whether the node has been visited before. With the knowledge of the former trajectory, the agent can produce a more informed policy. St also includes the utility u, which is referred to as the number of observable fron- tiers. These frontiers represent the areas with the potential to the target, which are generated at the boundary of the observed and unknown areas. St of each node in V2 are formulated as {17, 8,, u.}.
It is worth noting that in large-scale environments, Vú would be populated densely. Meanwhile, the information contained within them is sparse. Therefore, we implement a graph rarefaction algorithm (pseudo code in Appendix A) to prune irrelevant nodes and extract key edges of the viewpoints. Specifically, graph rarefaction first clusters the non-zero utility node set U into multiple groups according to the threshold radius dea (30 in practice). Then, the algorithm uses A* to search for the shortest path (, from the robot's current position Pt each group. After that, the refined node set V C U is constructed by waypoints on Ç, which is either out of deh or out of line of sight. The computation complexity is O(M + N K where M is the number of non-zero utility nodes, N the number of nodes chosen to compute the A path, K the number of edges for each node, and d the number of nodes on the resulting path. After the graph rarefaction process, the refined viewpoint graph G = E") would represent the complete information in the current robot belief. Finally, the G along with the corresponding planning states S are concatenated as the input of the policy network, i... the refined observation o, (G",
Action A: The collision-free graph extends incrementally along with the update of the agent's s observations Or for every decision step. Our context- aware policy network outputs the stochastic policy т for the agent to select the next waypoint among the neighboring nodes. Then, the agent updates its partial map while moving to the next waypoint.
Reward R: To promote efficient navigation, the agent receives a reward consisting of three parts.
The first part rs is constant time step penalty Ta (-0.5 in practice). The second partrs d(st-1)= d(s:) provides continuous feedback on the robot's proximity to the target destination, where d(s) is the distance between the agent current position jr: and the destination location computed by A* rb is used to encourage the agent to reach the destination as fast as possible. The last part ry is a fixed finishing reward, set to 20 while reaching the destination and otherwise. To sum up, the overall reward is ry (ot, at) = rs + c6 rb + ry, where e is a scaling parameter (c, = 1/64 in practice).
3.2 Policy Network
Inspired by [30, 31], we design attention-base based neural network (shown in Fig. 2) to sequence efficient movement decisions towards the destination. In our policy network, the encoder embeds the overall information of the current partial map, and the decoder utilizes the learned global features to reason about the dependencies between the current node and its neighbors and finally output an action policy (probability distribution over neighboring nodes).
The refined observation is first normalized and projected into d-dimensional (128 in prac- tice), termed ht. The node features ht are then passed into multiple attention layers (6 in practice) to aggregate the spatial representation of the current observation. The input of each attention layer consists of query vector 4 and a key-value vector k.v of the same dimension. The attention layer updates the query vector with the weighted sum of the value, where the attention weight depends on the similarity between the query and key. In each attention layer, the query qi key k; and value v; are first calculated as q: = @ h4. , k; W // WV E Raxd are learnable matrices. Next, the similarity between the query ga and the key ky is com- -.
Encoder:
1,5."" and U; = WV h respectively, where W W
puted with scaled dot product as Mij enij
The attention weights aiy are then obtained using Vi
softmax function: a;j Σラ=ı i Finally, the output embedding from an attention layer, denoted as M/ is calculated as the weighted sum of value vectors as Uy: h = ーØー d.y Additionally, an encoder edge mask M is applied to prevent each node from accessing its neighboring features. The output of the encoder, which we term the encoded embeddings ht, provides condensed spatial information to the decoder.

---

Decoder: In the decoder, the encoded embeddings of the current node (termed current encoded embeddings, he) are first extracted from the encoder as well as its neighboring encoded embeddings (termed neighboring encoded embeddings, ht). Then the current encoded embeddings and encoded embeddings are fed into an attention layer with h fi h lt This layer calculates output attention weights, representing the relevance of each neighboring node to the current node. These attention weights are concatenated with h and projected back to the d-dimensional feature together, termed current decoded embeddings, lis. The current decoded embeddings incorporate information from neighboring nodes into the representation of the current node. Finally, the current decoded embeddings and the encoded embeddings of neighboring nodes are fed into pointer layer [32], which is an attention layer that directly uses the normalized attention weights as output.
3.3 Training Settings
Inspired by Chen al. [33], we implement random dungeon map dataset for training, where there are total of 800 maps (each map is 1000 x 1000 pirels) like figure 3. To build the updated collision-free graph, the agent treats the points in the known free area as candidate viewpoints from 1600 points which are uniformly distributed to cover each dungeon map. Our model is trained using the Soft Actor-Critic (SAC) algorithm [34], where the maximum episode step is set to 128, and the size of the replay buffer is set 10000. The target entropy is set 0.01 log k. where represents the number of neighboring nodes. We the Adam optimizer optimize policy and critic networks with learning rates of 1 x 10-5 and 2 x 10-5 respectively. Our model is trained with one 19-10980XE CPU and one NVIDIA GeForce RTX 3090 GPU, and the training starts after collecting 2,000 steps in the replay buffer. We utilize Ray [35], distributed framework for machine learning to parallelize data collection. Training requires around 12 hours to converge.
workstation
4 Experiments
We set timeout, i.e., maximum decision steps (128 in practice), to prevent infinite navigation scenarios during testing, and failure if it exceeds this limit. To obtain plete picture, we report the main performance metrics for mapless navigation, including the av- erage success rate S(p), average travel distance D(m), and average travel time T(s) (including failed cases for both latter ones). In particular, T(s) the sum of each algorithm's 's step planning time (s) and the robot's resulting motion execution time (s)... We first compare with state-of-ll state-of-the-art conventional baselines in numerous dungeon environments (Fig. 3). Then, compare our model, some variants of our model, and the FAR planner (referred to as "FAR"" [24] in large-scale Gazebo environment (Fig. 5). Lastly, we deploy our trained model on hardware in real-world scenario (Fig. 6).
is considered
Simple
Medium
Complex
Figure 3: Examples scenarios with different complexity, showing the occupied area (grey cells). free space (white cells), start points (yel- low block), and target destination (red block).
approach
4.1 Comparisons in dungeon environments
To ensure fair comparison, we create random set of testing environments using the random dun- geon map generator [33], which were never seen by our trained model. Fig. 3 shows the diverse complexity of testing environments, which can be categorized into three types (50 scenarios each), noted as simple, medium, and complex. We define the quantitative criteria of the scenario complex- ity as follows: (i) The Fuclidean distance between the start point and target destination in each sce- nario. (ii) The overall number of connecting corridors in all rooms. (iii) The number of intersections that will be encountered along the assuall-optimm path. We consider two search-base approaches, LPA [3]. D* lite [4], as well as two sampling- based approaches: RRT [16], and BIT [23].
5

---

Table 1: Comparison results with state-of-the-art baselines in simple, medium, and complex scenarios (50 scenarios for each test set, standard deviation in parentheses), Environments are randomized 200 x 200 m² dungeons, the LiDAR's scanning range is 30 m, the robot's constant velocity 1.0 m/s, and the graph connectivity parameter 20 for our model.
Criteria
Ours
D* lite
LPA*
RRT
BIT
D(m)
224(#72)
214())8)
293(+116)
398(+208)
275(+166)
S
T(s)
(85
383(#152)
414(+:206)
(62
404(+:237)
Ts(s)
0.20(+0.08)
1.24(+0.56)
0.74(+0.47)
0.67(+0.54)
0.55(++0.46)
S(p) 100%
100 %
100%
94%
98%
D(m)
259(+83)
233()
383(+130))
497(2
360(+152)
M
T(s)
329(=105)
484(=87)
594(1298)
852 (156))
563(42285)
(s(s)
0.18(+0.12)
1.52(+0.73)
0.79(+0.55)
0.68(+0.48)
0.53(+0044)
S()) 100%
98%
94%
92%
98%
D(m)
375
(49())4)
466(+1663
544(++222)
493(+)215)
T(s)
477
151)
680(#221)
754(+265)
1063(=782)
806(+407)
(s(s)
0.22
1.66(=
0.83(+0.60)
0.62
0.
S(p) 98%
94%
88%
86%
90%
Ours
D* lite
LPA* *
RRT (partial)
BIT (partial)
Figure 4: Trajectory visualization in a representative complex scenario. The blue line is the robot's trajectory starting at the yellow dot and ending at the red dot. The green lines RRT and BIT are sampling trees in known areas, which are not included in D(m).
Evaluation results are reported in Table 1. For search-based approaches, model outperforms D* lite in terms of S(p). T(s), and Tp(s). Despite D lite showing lower D(m), our model still surpasses terms of T(s) by 15-32%, primarily due to the significantly shorter (s),, which is reduced by 83-88% D* lite is known for finding ncar-optimal paths, by searching and assessing node priorities incrementally. However, our results illustrate that D* lite results in high computation times (the worst (s) among all approaches), which prevents its use for real-tine planning. More- over, model outperforms I.PA* in three criteria by substantial margin of approximately 23%, 45%, and 73%. For sampling-basee approaches, our model surpasses RRT and BIT in all criteria. Our model finishes the task more than 30-41% 41% % faster than BIT, and more than 56-61% % faster than RRT. Additionally. our model demonstrates superior performance in terms of ((m), with improve- ments ranging from 18-28% to BIT and 31 31-49% to RRT. There, our results illustrate that both RRT and BIT are prone to generate inefficient trajectories, which consequently increases D(m) and T(s). Furthermore, we also conduct ablation experiments (reported in Appendix B) regarding the graph rarefaction algorithm and design of the encoder.
Fig. 4 shows an example where our planner generates more efficient trajectory, while other base- lines suffer from the misleading placement of the target destination. There, our model can be seen to exhibit more interest in unknown areas, with a higher drive toward the destination. We believe that this strategy substantially reduces the likelihood of aimless exploration and consistently aids in generating superior navigation trajectories.
4.2 Comparisons in Gazebo environment
We evaluate our planner and some variants (see Table 2) in highly convoluted environment charac- terized by narrow corridors and various obstacles. The robot is required to navigate through series

---

Table 2: Evaluations of our model and baselines in large-scale, complex Gazebo environment (130 m 100 m). We conduct these experiments to evaluate the performance of our model and FAR [24] (An efficient planning framework capable of handling path planning in unknown envi- ronments). The robot's constant velocity is 2.0 m/s, s. the LiDAR's scanning range is 30 m, and the D(m) and Ts) are the overall values after traveling to 7 successive goals. Our models were trained with k = 20, and used as (no extra training) for the k = 5, 10 tests.
Ours
FAR
Ours (no GR)
Ours (I: = 10)
Ours (k 5)
((m)
1367.2
2060.1
1483.5
1849.2
2103.6
T(s)
850.7
1359.4
1131.5
1112.1
1382.0
(p(s)
0.83(400.37)
2.42 (10.35)
1.96(10029)
0.38(+0.11)
0.36(10.12)
3
(Slan1)
(a) Our planner
(b) FAR
Figure 5: Trajectories comparison of two planners in large-scale indoor Gazebo environment.
of predefined target points successfully, similar to the setting presented in FAR [24]. To ensure fair- ness and consistency in the evaluations, we reset the planners after reaching each target point, which ensures each navigation task begins in a fully unknown environment.
As shown in Fig. 5(a) and 5(b), our model is capable of doing more informed exploration and gen- erating efficient trajectories towards the target destinations. Our experimental results (see Table 2) demonstrate that the significant reduction achieved by our model terms of D(m) and T(s), by up 692.9m and 508.7s, when compared FAR. It provides evidence that our model excels in gener- ating more optimal paths than FAR in unknown environments. Additionally, our model outperforms FAR in terms of T (s), needing only 0.83s per planning step.
In addition, we conduct further comparisons among several variants of our model, where we vary the number of neighboring nodes (20, 10, and respectively). Our experimental results indicate that = 10 surpasses FAR in criteria, without additional training. Furthermore, (s) with k less than that with to 254.4m, indicating much less efficient trajectories with A 5. Thus, it is not wise to set smaller value due to too sparse connectivity between nodes. To evaluate the significance of graph rarefaction, we test a variant of our model without sparsification, which results in more than 136% (s) and 8% ((m) compared to k = 20. Therefore, we believe that graph rarefaction significantly benefits our model.
10 by only 0.02s, while the D(m) of i: 5 is larger than k 10 up
4.3 Hardware Validation in a Real- World Scenario
Our ground vehicle utilizes Leishen C16 LiDAR for localization and mapping (shown in Fig. 6). The real-world scenario is 60 x 15 m² laboratory with randomly placed obstacles (e.g.. chairs, boxes, and camera tripod). The vehicle should travel through series of predefined points by fol- lowing the waypoint published by model. After receiving waypoint, the local planner [36] generates real-time and feasible motion commands for the ground vehicle. The experimental trajec-
7

---

5
2
bLEnd)
(Sart
A
Ground Vehicle
Figure 6: Validations in real-world environment. The ground vehicle, provided with partially observed point-cloud data, starts at point O and subsequently traverses series of consecutive points.
tory depicts a high-quality solution, which validates our model's effectiveness and shows promising applicability for real-world environments.
Limitations
The limitations of our method mainly revolve around adaptive sampling and smooth motion control:
We currently sample the observations uniformly, which may not precisely capture the spa- tial representation of the environment and leads to sub-optimal navigation strategies. To tackle this, future work will develop an online adaptive sampling strategy.
Our model currently plans the next waypoint under the assumption that the robot is omni- directonial (i.e., has no motion constraints), which may make reaching this waypoint diffi- cult in practice. The incorporation of a local motion planner (and its use to inform/train the policy) may facilitate navigation in unknown environments with dynamic obstacles.
6 Conclusion
In this paper, we propose a context-aware DRL. framework for mapless navigation that allows a robot to build a context of its entire partial belief over the environment, to infer the shortest route towards target destination. Our model achieves high-quality decision- makiin, especially in complex and large-scale environments, where it allows the robot sequence short-term movement decisions informed by global information about known areas. We also propose a graph rarefaction algorithm to filter out redundant nodes and corresponding edges in the graph input of our neural network, towards deployment in large-scale environments. We empirically demonstrate that our model outperforms state-of-the-ar baselines terms of average travel time and average planning time, with powerful generalizability to complex unknown environments never seen during training. Finally, we validate our approach in high-fidelity Gazebo simulations as well as on hardware, revealing promises for robotic deployments in the real world without further tuning.
Future work will first focus on the construction of local planner with more considerations kine- matic/dynamics constraints. Then, we will extend our framework to multi-agent mapless navigation, where robots need to reason about each other and plan efficient paths cooperatively, by leveraging synergies and avoiding redundant work.
Acknowledgments
This work was supported by Temasek Laboratories (TL@NUS) under grant TL/FS/2022011

---

References
[1] Bucki, J. Lee, and M. W. Mueller. Rectangular pyramid partitioning using integrated depth sensors (rappids): A fast planner for multicopter navigation. IEEE Robotics and Automation Letters, 5(3):4626-4633, 2020.
[2] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
02971, 2015.
[3] S. Koenig, M. Likhachev, and D. Furcy. Lifelong planning a* Artificial Intelligence 155(1-2): 93-146, 2004.
[4] Koenig and M. Likhachev. D** lite. Aaai/iaai, 15:476-483, 2002
[5] E. Hart, Nilsson, and Raphael. formal basis for the heuristic determination of minimum cost paths. IEEE Transactions Systems Science and Cybernetics, 4(2):100-107, 1968. doi: 10. L10//TSS...98.30000000
[6] P. Raja and S. Pugazhenthi. Optimal path planning of mobile robots: A review. International journal of physical sciences, 7(9): 139-1331,, 2012.
[7] J. D. Gammell, T. D. Barfoot, and S. S. Srinivasa. Batch informed trees (bit*); Informed asymptotically optimal anytime search. The International Journal of Robotics Research, 39 (5):543-567, 2020.
[8] Zhang. J. T. Springenberg. J. Boedecker, and W. Burgard. Deep reinforcement learning with successor features for navigation across similar environments. 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2371-2321 1-278.. IEEE, 2017.
[9] M. Pfeiffer, M. Schaeuble, J. Nicto, R. Siegwart, and C. Cadena. From perception to decision:
A data-driven approach to end-to-end motion planning for autonomous ground robots. In IEEE International Conference on Robotics and Automation (ICRA), page 1527-1533, IEEE, 2017.
[10] L. Tai, G. Paolo, and M. Liu. Virtual-to-real deep reinforcement learning: Continuous con- trol of mobile robots for mapless navigation. In 2017 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 31-36. IEEE, 2017.
[11] X. Xiao, Liu, G. Warnell, and Stone. Motion planning and control for mobile robot navigation using machine learning: survey, Autonomous Robots, 46(5):569-597, 2022.
[12] E. W. Dijkstra. A note on two problems in connexion with graphs. In Edsger Wybe Dijkstra: His Life, Work, and Legacy, pages 287-290. 2022.
[13] M. Likhachev, D. I. Ferguson, G. J. Gordon, A. Stentz, and S. Thrun. Anytime dynamic a#; An anytime, replanning algorithm. In ICAPS, volume 5, pages 262-271, 2005.
2005
[14] A. T. Le, M. Q. Bui, T. D. Le, and N. Peter. D lite with reset: Improved version of d lite for complex environment. In 2017 First IEEE International Conference on Robotic Computing (IRC), pages 160-163. IEEE, 2017.
[15] D. Ferguson, M. Likhachev, and A. Stentz. guide to heuristic-based path planning. Pro- ceedings of the international workshop on planning under uncertainty for outonomous systems, international conference on automated planning and scheduling (ICAPS), pages 9-18, 2005.
[16] M. LaValle al. Rapidly-exploring random trees: A new tool for path planning. 1998.
[17] S. Karaman and E. Frazzoli. Sampling-based algorithms for optimal motion planning, The international journal of robotics research, 30(7):846-894, 2011.

---

[18] J. J. Kuffner and S. M. Valale. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 2, pages 995-1001. IEEE, 2000.
[19] A. Perez, R. Platt, G. Konidaris, L. Kaelbling, and T. Lozano-Perez. Lqr-rit: Optimal sampling-based motion planning with automatically derived extension heuristics. In 2012 IEEE International Conference on Robotics and Automation, pages 2537-2542. IEEE, 2012.
[20] A. H. Qureshi and Y. Ayaz. Potential functions based sampling heuristic for optimal path planning, Autonomous Robots, 10:1079-1093, 2016.
[21] Wang, W. Chi, M. Shao, and M. Q.-H. Meng. Finding a high-quality initial solution for the algorithms in 2d environments. Robotica, 37(10):167711694 2019.
[22] J. D. Gammell, S. S. Srinivasa, and T. D. Barfoot. Informed rrt: Optimal sampling- based path planning focused via direct sampling of an admissible ellipsoidal heuristic. In 2014 JEEE/RSJ International Conference on Intelligett Robots and Systems, pages 2997-3004. IEEE, 2014.
[23] J.D. Gammell, S. S. Srinivasa, and T. D. Barfoot. Batch informed trees (bit*): Sampling -based optimal planning via the heuristically guided search of implicit random geometric graphs. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 3067-3074, 2015. doi: 10. 100////A
[24] F. Yang. C. Cao, H. Zhu, J. Oh, and J. Zhang, Far planner: Fast, attemptable route planner using dynamic visibility update. In 2022 IEEE/RS) International Conference on Intelligent Robots and Systems (IROS), pages 9-16, 2022. doi:10. (199//RO04771222222229999999
[25] J. Jin, N. M. Nguyen, N. Sakib, D. Graves, H. Yao, and M. Jagersand. Mapless navigation among dynamics with social-safety-eee awareness: saeeaaaeaeetya a reinforcement learning approach from 2d laser scans. In 2020 IEEE international conference on robotics and automation (ICRA), pages 6979-6985. IEEF, 2020.
[26] Y. F. Chen, M. Everett, M. Liu, and J. P. How. Socially aware motion planning with deep reinforcement learning. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1343-1350. IEEE, 2017.
[27] M. Everett, Y. Chen, and How. Motion planning among dynamic, decision-making agents with deep reinforcement learning. In 2018 IEEE/RSJ International Conference on In- telligent Robots and Systems (IROS), pages 3052-3059. IEEF, 2018.
[28] Gao, W. Ye, Guo, and Z. Li. Deep reinforcement learning for indoor mobile robot path planning. Sensors, 20(19):5493, 2020.
[29] G. Sartoretti, J. Kerr, Y. Shi, G. Wagner, T. S. Kumar, S. Koenig, and H. Choset. Primal: Pathfinding via reinforcement and imitation multi-agent learning. IEEE Robotics and Automa- tion Letters, 4(3):2378- 2019.
[30] Y. Cao, Y. Wang, A. Vashisth, H. Fan, and G. A. Sartoretti. Catnipp: Context-aware attention- based network for informative path planning. In Proceedings of The 6th Conference on Robot Learning, volume 205, pages 1928-1937, 2023.
[31] Y. Cao, T. Hou, Y. Wang, X. Yi, and G. Sartoretti. Ariadne: A reinforcement learning approach using attention based deep networks for exploration. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 10219-10225, 2023. doi:10.1 109/CCA44899
[32] O. Vinyals, M. Fortunato, and N. Jaitly. Pointer networks. Advances in neural information processing systems, 28, 2015.
10

---

[33] F. Chen, S. Bai, T. Shan, and B. Englot. Self-learning exploration and mapping for mobile robots via deep reinforcement learning. AIAA Scilech Forum, page 0396, 2019.
[34] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[35] Moritz, R. Nishihara, Wang, A. Tumanov, Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. Jordan, al. Ray: A distributed framework for emerging applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 561-577, 2018.
[36] Zhang, C. Hu, R. G. Chadha, and S. Singh. Falco: Fast likelihood- collision avoid- ance with extension human-guided navigation. Journal of Field Robotics, 778):1300-1313 2020.

---

Appendix
A Graph Rarefaction
Algorithm 1: Graph Rarefaction Algorithm
Input: non-zero utility node set U, map M, robot position Pt, threshold radius deh Initialize refined node set /" +- U, covered node set U 0
for E U do
if U then continue;
Find nearby node set in deni for of N do
| if line(u, v') is collision free then Ú t- v';
end
Find path Ç from Pt to u, set ref node Oref = vi; for <o if tine(Uref) (i) is not collision free or L (Uref Si) 2 der then
Uref (i-1, V
Uref
end end
Output: collision free edge set E" based on V and M.
B
Ablation experiments
We carried out ablation experiments on our model to evaluate the impact of its key parameters. We first introduced the various ablation cases, which are our model with the original setting (termed Ours), our model with original viewpoint graph (i.e., without graph rarefaction, termed Ours (no GR)), and our model with encoder layers (termed Ours (Encoder-2)), All the cases are evalu- ated in 2D environments (same test set as in the paper) and reported in Table 3.
Table 3: Ablation experiments on our model in simple (S), medium (M), and complex (C) scenarios (50 maps per test set). The comparison metrics are average travel distance ((m). motion execution time T (s)), step planning time (s),, and test success rate per scenario Syy). respectively. The standard deviation of the comparative metrics is indicated by the parentheses.
Criteria
Ours
Ours (no GR)
Ours (Encoder-2)
((m)
224(1122
245(=55)
382( = 198)
s
T(s)
285
(13(1101)
489( 253)
T2(3) 0.20(+0.088
0.34(#0.06)
0.18( +0.10)
S(p) 100%
100%
98%
D(m)
259( (+)
259
276(+104)
443(=233)
M
T(s)
329 105)
452(
561(-294)
Ty(s)
0.18
10.11)
0.31
0.13(10.03)
S(p) 100%
100%
88%
D(m)
375(+119)
410(+127)
611(=240)
C
T(s)
471(1151)
573 1235)
774 330))
(2(s)
0.22(110113)
0.38 (3.15)))))
0.16 10.04)
S(p)
98
96%
84%
As shown in Table 3, the model Ours (no GR) degrades slightly in terms of D(m), T(s), and S(p) compared to our original model. However, we note that our original model achieves computational efficiency that is twice as fast as the step planning time (s),,, which indicates the significance of graph rarefaction algorithm our approach. Moreover, we find that model Ours (Encoder- 2) exhibits a large drop in performance, particularly in the medium and complex scenarios. This result indicates that an encoder with more layers assists the robot in avoiding myopic decisions, particularly long-tern term mapless navigation tasks.
12


Reviews:
[{"review_id": "zHmdDdmA6q9", "reviewer": "Reviewer_qPjV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a learning-based framework for autonomous robotic navigation in unknown areas. The framework employs a context-aware policy network to extract and utilize spatial information to generate an effective path planning policy. Additionally, a graph rarefaction algorithm is proposed to remove irrelevant or redundant information in the navigation environment. The approach allows the agent to learn an efficient path planning policy, which can re-plan at low latency while receiving new environmental observations. The approach is evaluated on hundreds of test scenarios and compared to benchmark planners (D*lite and BIT). Results show that the approach reduces average travel time (i.e., makespan) up to 61.4% and average planning time up to 88.2%. The approach is also validated in high-fidelity Gazebo simulation as well as on ground robot, highlighting its promising applicability to real-life environments without further tuning. ", "summary_of_recommendation": "This is a solid technical report of the authors' recent work on incorporating reinforcement learning and graph representation for navigation. Although there are already plenty of works in this field. Both the experiments and paper organizaiton are convincing, although the paper lacks some of the important comparisons with other RL based navigation works and ablation of their own designs. In my opinion this paper is around the borderline.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The authors should add a table of ablation expeiments over their own method to demonstrate the effectiveness of their design.\n\nMissing some of important notations, for example what are the red dots in Figure 2? Also it would be super helpful if the captions explain more major details and ideas of the figures.\n\nIn table 2, should be included in caption what is FAR and citation.\n\nIn the real world experments shown in the video attachment, the robot never makes a mistake at any intersection even if the goal is invisible. I wonder why this happens? Are there any cases where the robot makes mistakes, and if so I'd recommend adding some of them to the attachment to avoid confusion.\n\nFinally, larger scale map navigation is encouraged to study and add a table of how the performance is like for various tasks of different challengeness.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed approach is an effective method for mapless navigation in unknown environments. The results demonstrate that it is capable of reducing average travel time (i.e., makespan) up to 61.4% and average planning time up to 88.2% when compared to benchmark planners (D*lite and BIT). Additionally, the approach is validated in high-fidelity Gazebo simulation as well as on ground robot, highlighting its promising applicability to real-life environments without further tuning. The encoder embeds the overall information of the current partial map, and the decoder reasons the local information of the current node and its neighboring nodes to finally output the action policy. The policy network is an attention-based neural network that is used to generate efficient path planning towards the target.\n\nThe proposed model appears to be effective in generating more efficient paths than the conventional baselines in unknown environments. However, it is unclear if the model is able to handle dynamic obstacles, which may be encountered in real-world scenarios.\n\nThe proposed model has been tested and validated in a large-scale Gazebo environment, and the results demonstrate that the model is able to generate more optimal paths than the FAR planner in unknown environments. However, the model may not be able to handle very large-scale environments, and further testing is needed to evaluate its performance in such scenarios.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "bX1IucK5yHc", "reviewer": "Reviewer_cs7G", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a new DRL framework for navigating unknown environments. The environment is modeled as a graph that is continuously learned. To mitigate the large graph space, a pruning method is used that improves path planning efficiency. This graph is used by a neural network policy, learned using DRL, to learn planning policies. The developed  algorithms outperforms related motion planners in terms of runtime and traveled distance.", "summary_of_recommendation": "Overall, the paper proposes an interesting approach to solve navigation problems in unknown environments. However, the comparative experiments are not convincing. Additional experiments are needed to demonstrate the benefits of this methods over existing ones.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1) Why is it called \"context-aware\" DRL? Is it because of the graph rarefaction algorithm? The authors need to explicitly explain what \"context\" means, how it is utilized, and what are its benefits.\n\n2) What does mapless navigation mean? Does it just mean that the environment is unknown or it additionally implies that planning is performed in the unknown environment without actually learning the map of the environment (e.g., an occupancy grid map)? \n\n3) Although the literature review is complete, it is unclear if the proposed method really address the existing challenges. For example, it is claimed that sampling based methods are suboptimal, unstable, and they may get stuck in local minima  (although asymptotically they will find a feasible solution). Although this is true, several methods have been proposed to overcome such challenges e.g., by developing intelligent sampling strategies [A], [B] or hallucinating the unknown environment [C]. Such methods are not discussed. At the same time, the proposed does not necessarily address the above challenges (even though the the empirical performance is satisfactory). The authors do not need to cite these specific papers but they need to explain better how this work fits in the existing planning literature\n\n[A] \"Perception-based temporal logic planning in uncertain semantic maps.\" IEEE Transactions on Robotics 38.4 (2022): 2536-2556.\n\n[B] \"Guided Sampling-Based Motion Planning with Dynamics in Unknown Environments.\" arXiv preprint arXiv:2306.09229 (2023).\n\n[C] \"Map-predictive motion planning in unknown environments.\" 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.\n\n4) Similarly, the authors claim that existing learning-based planners cannot be applied to large and unseen environments. This is not entirely correct. Most existing learning-based planners do not have any provable scalability/generalization guarantees but they have quite impressive empirical performance. See e.g. [D]. As before, the authors do not need to cite this specific paper. However, a more fair presentation of existing works is needed.\n\n[D] \"Socially aware motion planning with deep reinforcement learning.\" 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017. \n\n5) In Section 3.1., is the POMDP state and action space discrete? Does every time the robot knows its current state, i.e., the state s it belongs to? In general, it's a bit unclear how the POMDP models an unknown environment and what exactly is unknown in the environment. E.g., is the state transition function T known?\n\n6) What assumptions are made about the observation function? Is it assumed to be a perfect observation/sensor model? I.e., is its output always correct? How can it implemented in practice?\n\n7) In Section 3.2, what exactly is the input to the neural network policy? Is it the graph Gt and the current robot state?\n\n8) What is the target entropy in Section 3.3? How is it used?\n\n9) In the comparative experiments, the traveled distance and travel time are used as evaluation metrics. The authors should mission success rates as well. This is a much more important metric than runtimes and traveled distance. In fact, it is not surprising that a learned policy will be faster and more efficient than a traditional planning policy (e.g., RRT). Also, are the reported results based on the average performance of multiple runs or a single run? If the former, in the reported travel times/distances, are all runs considered, or only the ones where the learned policy succeeds?\n\n10) In figure 4, why do the blue paths for D* and LPA* initially move quite far away from the target? That result looks somewhat incorrect given that these methods treat the unknown space as free space. Also, how are the baselines executed? Is replanning triggered at every single iteration or only at time steps where the current paths intersect with unexpected obstacles? This is critical to fairly evaluate the runtimes of these methods; note that replanning at every time step may not be necessary. \n\n11) How does the proposed algorithm perform against other learning-based planners? Does it outperform them in terms of scalability and generalization to unseen environments?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n+ The paper is in general well written and seems to be technically correct\n+ It performs better than traditional planners (e.g., RRT) in terms of runtime and traveled distance\n\nWeakness:\n-  The comparative experiments are not convincing. Comparisons with respect to mission success rates are missing, which is probably a much more important metric than runtimes. In fact, it is not surprising that a learned policy will be faster and more efficient than a traditional planning policy (e.g., RRT). This is critical as the paper claims that existing planners lack of stability and optimality but there is no evidence that the proposed method addresses these shortcomings.  \n- Comparisons against other learning-based approaches is missing. The authors claim that existing method generalize poorly to new environments. This is in general an issue in related works but there are no comparisons showing improvement over related works. \n- The novelty of the paper compared to related works is vague.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "T1oSivnv8XA", "reviewer": "Reviewer_eMFY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a deep-reinforcement-learning method for guiding a mobile robot to reach a predefined target in an unknown environment.  The proposed method is evaluated against state-of-the-art methods in extensive simulator experiments.  The proposed method is observed to significantly reduce planning and travel time in the simulator experiments.  Also, a hardware demonstration is presented, without comparison with state-of-the-art methods.", "summary_of_recommendation": "The paper focuses on a fundamental robotics and learning problem, proposing a method that appears promising for real-world deployment. In my opinion, the current version of the paper would benefit from (i) discussing corner cases, (ii) clarifying how the proposed method can be trained for real-world deployment, and (iii) providing supporting evidence of its superiority during real-world deployment.\n\nUPDATE: I keep my original recommendation (Weak Accept) upon accounting for the generalization weaknesses of deep learning methods.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "Please see my comments in the weaknesses section above.  Also, for the hardware experiment, what is the training set?  Is it the same as for the 2D-map synthetic experiments?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n\n-  A well-written paper that is a pleasure to read.\n-  The problem is well-motivated and of relevance to the CoRL community.\n-  Extensive evaluation against state-of-the-art methods in synthetic datasets.\n\nWeaknesses\n\n-  In the discussion of the result from the hardware experiment, it is stated that the robot ends up following a near-optimal solution.  I would justify why this statement holds true: since no comparison with other methods is included as part of the experiment, it appears hard to infer how near-optimal the solution is, and whether the solution's quality is due to the used local planner or due to the context-aware part of the proposed method.  Is it the local planner that primarily enables the resulting trajectory or the context-aware part of the proposed method?\n\n-  Since the proposed method is based on deep learning, I would say that one can find corner cases where the method fails.  Although the current synthetic experiments suggest good generalizability of the method, could a discussion be included in the paper discussing corner cases and potential remedies?\n\n-  I would make explicit early in the paper what the \"context\" is and how it is integrated into the algorithm.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "KfwX0UqnYR", "reviewer": "Reviewer_CUZW", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this work, the authors describe and implement a method for leveraging deep reinforcement-learning (DRL) for efficient global path\nplanning in environments lacking a prior map (mapless navigation). The proposed method is touted for its improved performance over\nclassical path-planning methods, in terms of both required computation time and efficiency (shortness of distance) of planned paths.  The\nsystem is trained on simple, randomly-generated 2D dungeon-map environments, and is evaluated over new 2D maps, a Gazebo simulation environment, and a real-world office-like environment.", "summary_of_recommendation": "In summary, this work presents a reasonable and well-motivated algorithm for path-planning using deep reinforcement learning, but lacks sufficient rigor in embedding itself in the prior literature. It furthermore is limited in the scope of its experimental findings/analysis, and in its interrogation of/motivation for replacing classical baseline algorithms with a learning-based approach.\n\nBeneath these shortcomings I believe there could be a valuable contribution to the robotics literature.  Additional focus and clarification lent to these points would do well to form this paper into a much stronger candidate for submission.  \n\nPost-rebuttal comments: I would also like to note that the above ratings accurately reflect this reviewer's views of the paper upon reading the authors' rebuttal responses.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "- Could you elaborate further on the embedding of this work into the established literature?  This would include more related work describing learning-based approaches to this task, as well as elaboration/clarification of the heuristic method taxonomy.\n\n- What is the computational complexity of the proposed graph rarification algorithm?  How does it compare to the complexity of some of the full baseline algorithms?\n\n- Can you include more specifics regarding the comparative implementations of the baseline algorithms (D*, RRT, etc.)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n\nStrong Apparent Performance Characteristics\nThe comparative results between the proposed DRL method and classical methods in the literature displays the utility of learning-based methods in providing reasonable solutions to what would otherwise be complex, time consuming queries.  The model is shown to be\nsubstantially faster than implementations of classic and state-of-the-art heuristic methods (though that comparison would benefit from a more detailed account of software used, so as to confirm that the performance is not limited by the implementation).  \n\nDescriptive Figures\nThe figures provided are useful for clarification and reader understanding of the evaluated environments, as well as the proposed architecture (Fig 2).\n\nMulti-Environment Evaluation\nThe authors test their method across three environment types/fidelities, including a real-world experiment\n\n\nWeaknesses\n\nWeak Taxonomic Overview and Embedding in the Literature\nIn their introduction and discussion of background/related works to that which is proposed here, the authors offer a high-level taxonomy\nof approaches to robotic path-planning.  While the trichotomy of search-based, sampling-based, and learning-based approaches described is a valid breakdown of path-planning methodologies, the arguments offered for the shortcomings of previous works in each set of high-level approaches are limited in their persuasiveness.  For example, the array of varying optimal and near-optimal shortest path\napproaches are dismissed due to replanning \"always\" incurring high computation costs, as well as a lack of robustness and cognitivity\nfrom methods which leverage a heuristic function.  The dismissal of traditional sampling-based methods is equally limited, citing limitations of sampling-based approaches that are by no-means universal (oversampling in free and unknown areas).  Further, it is stated that random-sampling in general yields instability and suboptimality in pathing, despite already citing asymptotically-optimal sampling approaches, and ironically citing that claim itself from the LQR-RRT* paper; the body of which is a clear contradiction to this claim.\n\nUnclear Preprocessing Complexity and Utility\nIn transforming the current map into the agent observation, the authors propose a graph simplification algorithm with aim to avoid\nblowup in space-complexity as the scale of the environment is increased (as the viewpoint graph is otherwise a dense 2D-grid).  It\nwould be helpful to include comparisons between the proposed algorithm and the corpus of existing work in the literature of graph simplification for path-planning.  Furthermore, it is unclear from the paper the computational complexity/overhead incurred by this algorithm at each planning step.  The justification for applying learning to the problem domain is weakened when the preprocessing step to that learned system is itself nontrivial.  The reader would further benefit seeing evaluation of baseline planning methods operating directly over this refined viewpoint graph, rather than a general map space.  (If it is the case that the viewpoint graph is used in the evaluations of baseline search-based algorithms, it is not made clear in the paper as it stands).  \n\nPotential for Overfitting to Training Data\nWhen leveraging a learning-based approach to decision-making problems with partial state visibility, one must be deliberate in their efforts to acknowledge the potential for the agent to learn not only the desired task, but also non-generalizable patterns from the training\ndata/environments.  Leveraging exclusively generated dungeon maps for training, and comparing the agent's path-planning distance to\ngeneralized best-first search is unfair in the general case.  Through training, the agent can learn patterns present in the dungeon-map\ngeneration algorithm, and use this information to improve its  via encoded prior knowledge.  It would benefit the work to see comparisons\namong a wider array of simple environment topologies.  \n\nLimited Real-World/High-Fidelity Evaluation\nWhile the inclusion of evaluation in a realistic robot simulator and the real-world is beneficial to the paper, the breadth and depth of\nstudy in these environments is too limited to be sufficiently convincing to the reader.  The Gazebo environment simulation would\nserve as an excellent grounds for comparison of the traditional algorithms tested in the simple environment, yet only FAR is evaluated\nagainst.  Reviewing the paper which proposed the FAR Planner, the paper would benefit from at least citing the evidence of FAR's\noutperformance of the other baselines (though a full study with more baseline algorithms would be even stronger).  Furthermore, the\nreal-world results generated by the algorithm are presented without comparison, leaving the reader to evaluate the real-world efficacy of the proposed method purely qualitatively.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors  present a novel approach to address the challenge of mapless navigation in unknown environments using deep reinforcement learning. The paper is well-written and clearly motivates the problem, offering a detailed overview of the proposed methodology. The reviewers have provided valuable feedback and raised several critical points that need to be addressed for the paper's improvement. The authors' responses indicate a thorough understanding of the reviewers' comments, and they have proposed revisions that will significantly enhance the clarity, technical quality, and comprehensiveness of the paper.

Strengths:
The paper introduces an intriguing approach to mapless navigation in unknown environments using deep reinforcement learning. The problem is well-motivated, and the paper clearly outlines the benefits of the proposed approach. The technical quality of the work is evident, and the authors demonstrate a strong grasp of the theoretical foundations and practical implementation of their method. The paper is generally well-organized and clearly presented, making it accessible to readers from various backgrounds. The inclusion of both simulation and real-world hardware experiments further strengthens the validity of the proposed approach.

Weaknesses:
The weaknesses highlighted by the reviewers primarily revolve around the need for additional explanations, discussions, and experimental analyses. The lack of comparisons with other learning-based planners, the assumption of perfect localization and mapping, and the absence of corner cases are notable issues that need to be addressed. The concerns raised about the comparative experiments, the necessity of mission success rates, and the handling of search-based algorithms' replanning behavior should also be considered for improvement. Furthermore, clarity on the concept of "context-aware" reasoning and elaboration on the training process for real-world deployment are areas that require further attention.

Reviewer-Specific Comments:
Suggestions regarding embedding the work in the established literature, computational complexity, and more specifics on baseline implementations should be addressed in the revised manuscript.
Feedback about discussing corner cases, clarifying the training process, providing mission success rates, and addressing the absence of comparisons with other learning-based planners should be taken into account.
Additional questions about the POMDP formulation, observation function assumptions, policy network input, target entropy, and graph movement in Figure 4 need to be thoroughly addressed.

The authors' responses indicate a clear understanding of the reviewers' feedback and their commitment to improving the paper. The planned inclusion of more comprehensive discussions on related work, addressing the concept of "context-aware" reasoning, providing clarifications on the training process for real-world deployment, and incorporating a detailed explanation of the POMDP formulation are positive steps towards enhancing the manuscript's technical quality and clarity. The authors' efforts to address corner cases, provide mission success rates, and clarify replanning behavior for search-based algorithms are also commendable.
Decision: Accept (Poster)