HTRON: Efficient Outdoor Navigation with Sparse Rewards via  Heavy Tailed Adaptive Reinforce Algorithm
Authors: Kasun Weerakoon
Souradip Chakraborty
Nare Karapetyan
Adarsh Jagan Sathyamoorthy
Amrit Bedi
Dinesh Manocha
Venue: CoRL 2022 Poster (2022)
Keywords: Robot Navigation
Deep Reinforcement Learning
Heavy Tailed Policy

Abstract: We present a novel approach to improve the performance of deep reinforcement learning (DRL) based outdoor robot navigation systems. Most, existing DRL methods are based on carefully designed dense reward functions that learn the efficient behavior in an environment.  We circumvent this issue by working only with sparse rewards (which are easy to design) and propose a novel adaptive Heavy-Tailed Reinforce algorithm for Outdoor Navigation called HTRON. Our main idea is to utilize heavy-tailed policy parametrizations which implicitly induce exploration in sparse reward settings. We evaluate the performance of HTRON against Reinforce, PPO, and TRPO algorithms in three different outdoor scenarios: goal-reaching, obstacle avoidance, and uneven terrain navigation. We observe average an increase of 34.41% in terms of success rate, a 15.15% decrease in the average time steps taken to reach the goal, and a 24.9% decrease in the elevation cost compared to the navigation policies obtained by the other methods. Further, we demonstrate that our algorithm can be transferred directly into a Clearpath Husky robot to perform outdoor terrain navigation in real-world scenarios.

HTRON: Efficient Outdoor Navigation with Sparse Rewards via Heavy Tailed Adaptive Reinforce Algorithm
Kasun Weerakoon; Souradip Chakraborty', Nare Karapetyan, Adarsh Jagan Sathyamoorthy, Amrit Singh Bedi and Dinesh Manocha.
Abstract: We present a novel approach to improve the performance of deep rein- forcement learning (DRL) based outdoor robot navigation systems. Most, existing DRL methods are based on carefully designed dense reward functions that learn the efficient behavior in an environment. We circumvent this issue by working only with sparse rewards (which are easy to design), and propose a novel adaptive Heavy-Tailed Reinforce algorithm for Outdoor Navigation called HTRON. The key idea in this work is to utilize heavy-tailed policy parametrizations which implic- itly induces exploration in sparse reward settings. We evaluate the performance of HTRON against Reinforce, PPO and TRPO algorithms in three different outdoor scenarios: goal-reaching, obstacle avoidance, and uneven terrain navigation. We observe in average an increase of 34.41% in terms of success rate, 15.15% de- crease the average time steps taken to reach the goal, and 24.9% decrease the elevation cost compared to the navigation policies obtained by the other meth- ods. Further, we demonstrate that our algorithm can be transferred directly into a Clearpath Husky robot to perform real-world outdoor terrain navigation.
Keywords: Navigation, Deep Reinforcement Learning, Heavy Tailed Policy
Introduction
Autonomous robot navigation in complex outdoor environments has been an active area of research. The resulting navigation systems are used for different applications, including delivery [1], search and rescue [2), planetary explorations [3], etc. Such applications require robots to gather useful information efficiently from the environment to make intelligent navigation decisions [4]. To this end, Deep Reinforcement Learning (DRL) techniques have been widely employed recent robotic systems due to their inherent exploration and exploitation capabilities to gather necessary informa- tion from the learning environments [5, 6, 7, 8, 9, 10]. Nevertheless, major challenge for DRL, based algorithms is dealing with sparse rewards in continuous state and actions spaces. In particular, the rewards are sparse in navigation settings because it's only available on reaching a goal (positive reward) or hitting an obstacle (negative rewards) [11]. Hence, training DRL policies under sparse reward settings oftentimes lead to instabilities and convergence issues that can significantly degrade consistent performance, especially in navigation applications [12, 13).
To improve the training efficiency and deal with sparse rewards, two popular methods in the literature are reward shaping [14] and cimmonstrationguudee guided learning [15, 16). The primary objective of rewards shaping-based methods to induce an exploration strategy using curiosity-driven methods which provide an additional pseudo reward for exploration in the environment [17]. However, such methods require carefully designed intrinsic rewards which can introduce expert-specific bias the learning systems hindering the overall performance. Similarly, demonstration -ugiddd methods have a high dependence on expert supervision which could be difficult to obtain in practice for many outdoor navigation tasks [18]. Instead of modifying the rewards, a more pragmatic approach is to
"These authors contributed equally to this work. Kasun Weerakoon, Souradip Chakraborty, Nare Kara- petyan, Adarsh Jagan Sathyamoorthy, Amrit Singh Bedi and Dinesh Manocha are with the University of Mary- land, College Park, MD, USA. Email: (kasunw, schakra3, knare, asathyan, amritbd, dmanocha] @und. edu.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

try to work directly with the sparse rewards [19] and make the DRL based policies to train for the outdoor navigation task at hand. But the major challenge how to induce inherent exploration into the training without reshaping the rewards. Recent work in literature (20, 21] suggest that one way to handle these to use heavy-tailed policy (such as Cauchy) parametrization techniques. Motivated by these factors, in this work, we try to find optimal behaviors for outdoor navigation tasks while directly operating under sparse reward settings.
Main Contributions:
1.
We present heavy-tailed policy formulation with adaptive gradient tracking to address three navigation challenges encountered in complex outdoor environments (gaal-reaching, obstacle avoidance, and navigating on uneven terrains). Our proposed algorithm HTRON outperforms three state-of-the-art algorithms: Reinforce [22], PPO [23], and TRPO [24] in terms of cumulative reward return with improved sample efficiency for faster convergence.
2. We propose a set of novel sparse reward functions that do not require careful reward de- signs yet are capable of successfully navigating a differential drive robot using an improved stochastic policy gradient method.
3. We evaluate the navigation performance of HTRON both simulated and real-world com- plex outdoor environments using a Clearpath Husky robot. HTRON results in an increase up to 34.41% in terms of navigation success rate, 15.15% decrease in the average time steps taken to reach the goal, and 24.9% decrease in the elevation cost compared to the navigation policies obtained by the other methods.
2 Related Work
Learning-based approaches, in particular DRL have pushed the boundaries of mobile robot navi- gation for unstructured and dynamically changing environments [25, 26, 8]. Nevertheless, these methods suffer from sample efficiency under sparse rewards settings because dense rewards play an important role in learning for DRL methods. We summarize the related work next.
Navigation, path planning and obstacle avoidance are extensively- studied fundamental research problems in the field of robotics [27, 28]. Number of determinis- tic [29, 30] and stochastic [3] algorithms have been proposed in the past three decades. How- ever, robot navigation in complex outdoor environments still remains a challenging problem given the complexity and diversity of the setting. To perform robot-centric decision -aakiig,, deep rein- forcement learning (DRL) strategies have been employed in recent studies for outdoor navigation [33, 34]. For example, deterministic policies such as DDPG[26), A3C[35], and DQN[36] with dense rewards have been incorporated for robot navigation on uneven terrains. An adaptive model that directly learns control and environmental dynamics is presented in [8]. Morcover, scgmentation[37] and self supervised [25] methods for identifying navigable regions have also been utilized with a combination of DRL based navigation methods ensure stable and efficient navigation.
Outdoor Robot Navigation:
Sparse Reward Settings: In sparse reward settings, an environment rarely produces useful reward which significantly affects the reinforcement policy leaning [11]. To deal with this issue, several re- ward shaping methods based on intrinsic curiosity and information gain-based shaping have been proposed [38]. For example, reward shaping based methods such [39, 17] encourage the agent to explore unvisited states by modifying the actual reward output. However, such approaches re- quire additional effort and expertise in reward function design. Imitation learning and learning from demonstration methods are another approach for dealing with sparse rewards in practice [40]. How- ever, obtaining reliable expert demonstrations for continuous control tasks is practically infeasible and challenging [41, 15].
Heavy-tailed Policy Parametrization: The possibility of incorporating heavy-tailed distributions to RL tasks is theoretically analyzed in recent literature [42, 21]. For example, in [43], beta distri- bution is utilized with dense rewards for stochastic policy training to enhance the state exploration capabilities. Further, a heavy-tailed policy gradient method is proposed in [42] to find convergence to global maxima while minimizing the risk of local maxima convergence. Inspired by these ideas, we propose a heave-tailed adaptive reinforce algorithm deal with sparse rewards in complex out- door environments.
2

---

Problem Formulation and Our Approach
3.1 Outdoor Navigation via Reinforcement Learning
Mathematically, we formulate the outdoor navigation problem as Markov Decision Process (MDP) in continuous state and actions spaces:
M: P,
(1)
where S denotes the state space including distance from the goal, heading, roll and pitch angle; A the actions space (linear, angular velocity); P(s'|s, a) is the transition kernel; r(s, a) is the reward and y (0,1) denotes the discount factor. The objective of the robot is learn a navigation policy Te (als) parameterized by 0 (which controls the probability of taking particular action given state s) perform efficient outdoor navigation. This is achieved by solving the following optimization problem
max ((0) = V V=*(
where (so) = EE E - 7'r(s;, 5 5, Ta (-1st) is the average cumulative reward (or value function), and so denotes the initial state along trajectory {s,, az, r(5,, a,) Tmmo o- We note that since the goal of the outdoor navigating robot is to reach goal and make sure that the problem (2) is actually achieving that, we need to design the reward function r'ss, a) such that upon maximizing its cumulative value, our goal is achieved. This is of the major challenges in applying RL ideas to outdoor navigation. The majority of the RL results revolve around learning in standard environments and platforms like Gym. Mujoco [44] or other simulated environments where the rewards structures are already defined. However, in practical outdoor navigation scenarios, it is extremely hard to generate dense rewards due to the possibility of huge unknown trajectories a robot can take while navigating in the environment [34, 8]. The problem is exacerbated in continuous state-action spaces where one needs to define rewards even for infinite possibilities, which could be impractical.
3.2 Our Approach
To deal with this issue and empower the application of RI. to outdoor navigation, we take a different route and advocate the use of sparse rewards. The advantage of sparse rewards is the simplicity of their design as they need to be defined only at specific goals/sub-goals to the robot. However, the sparse reward makes the learning problem hard due the non-trivial estimation of value functions over the continuous state space, which can be mitigated through the use of heavy-tailed policy gradient algorithms{42. 21]. In this work, we start with construction of an experimental sparse reward design methodology for outdoor navigation scenarios and proceed with the description of our algorithm.
3.2.1 Outdoor Navigation using Sparse Rewards
We categorize the level of complexity in the reward structure using three navigation scenarios in outdoor environments. In all scenarios, our agent modeled as differential drive robot with two-dimensional continuous action space a = (v,w) (i.e. linear and angular velocities), whereas the state space dimension varies for each scenario. The state inputs are obtained real-time from the robot's s odometry, and LiDAR sensors. The list of state inputs used throughout the three scenarios is as follows:
dgoat E IR+ current distance between the robot and its goal;
Ogoaal [0, mr] current angle between the robot's heading direction and the goal;
(1-1, [-1,1] actions from the previous time step (i.e. linear and angular velocities);
(Oratt, (ptthh) ) E [0, .] roll and pitch angle of the robot respectively:
Dats E [0, 10] laser sean vector that includes distance to the obstacles around the robot (i.e. 360 laser scan data as a vector with ~ 720 elements) at a given time.
Scenario (Goal Reaching Baseline): In this scenario, the robot is placed in an obstacle-free outdoor terrain with an objective to reach given goal location. We incorporate distance to the goal, heading angle to the goal and previous actions to define a four-dimensional state space =

---

Rewaadd
Costicilo Difaaice Volaa
Heattigg Brgle (degirems)
Disttnce (m)
Dvschnce (mm)
(a)
(b)
Sparse reward surface visualization: Reward distribution with (a) dgoat, agoat and (b) dgoat, Dons- We observe that the sparse reward settings lead to chaotic and unstructured reward surfaces. These plots show that reward formulations in the navigation scenarios are significantly sparse and do not include uniquely identifiable maxima that can be reached smoothly.
Figure 1:
(dgoat, Ggoot, 0t-1: w]-1]. We utilize a set of sparse rewards that are easy to implement for policy training. For the goal reaching task, we define Theading and rdist rewards to maintain the robot's heading direction towards the goal and to encourage reaching the goal respectively. Hence,
39 2
dgout
Theading 1 (==/4: and Tdist
N
2 .og) + (0,
(3)
where Bg is constant used to adjust the reward amplitude for goal reaching, og E 0.1] is the variance of the normal distribution of, which maintain very low range. This ensures,
the robot receives reward rdist only when it reaches either Lpoa! or the defined goal. Otherwise, 2
rdist is zero. These reward formulations intuitively motivate the robot to reach the goal by rewarding it when the halfway or the complete distance to the goal is traveled. The total reward is obtained as, Ttor = Theading + Tdist- The reward distribution w.r.t the state parametric space is visualized in Figure ((a) to highlight the reward sparseness.
Scenario 2 (Obstacle Avoidance): In this scenario, the robot placed in an outdoor area with static obstacles such as trees, walls, and sharp hills. The objective is to reach the goal while avoiding collisions with these obstacles. Additionally laser scan vector is taken into account in the state space Agnat, gooat 1t-1, W1-1, Dobul. In addition to the two rewards defined in the previous scenario (i.e. Theading and rdist), we introduce an additional sparse reward robs as,
Tobs { -10 miminnee Dobe) I deollistonn 0
(4)
otherwise,
to penalize collisions. Here, deollisson is the minimum safety clearance that the robot should main- tain with obstacles. The modified total reward obtained as, I'ot Theading Pdist Tobs-
Scenario (Navigating on Uneven Terrains): this scenario, the robot is placed in a highly elevated outdoor terrain where it might flip over experience instability in some regions. The objective is to navigate to goal location while avoiding steep elevations in the environment. The corresponding state space is [dyoat, O'goat, Ut-1, W Oroti, Opitch)] To minimize navigating steep elevations, we incorporate another sparse reward term (Tstable) with the goal reaching reward pair. We consider the states where the robots roll (0rot) pitch angle (0patch) exceeds + द steep elevations (i.e. unstable robot orientations). Hence, the stability sparse reward is defined as,
Tstable 1 (10)0 L (10pnea|2=/4)
Hence, the total reward function for this scenario is obtained as, Ntot = Theading + Fdist + Fatable-
3.2.2 Heavy-Tailed Reinforce Algorithm for Outdoor Navigation (HTRON)
Just operating with sparse rewards is not sufficient to solve the outdoor navigation problem because the learning procedure won't it be able to explore the environment in required manner. For instance, continuous control robotics research primarily relies Gaussian policy parametrization given by Ta(a))) = (s) 8, 0,02) where controls the mean of the Gaussian, (ps)) denotes the states feature representation S - Rdd with d & g, and o² variance. However, the performance of Gaussian policy parametrization under sparse rewards is shown to suffer badly in the latest research

---

Algorithm Heavy-Tailed Reinforce for Outdoor Navigation (HTRON)
I: Initialize Initial policy network parameter Po. discount factor step-size 17, Adaptive momentum parameters 31. _2.. 6, and +
Repeat for & L....
2: Collect the trajectory (x(Ok) by utilizing policy Te, with robotic action driven constraints i.e the actions
are first sampled as a ~ no, (a]s) and then we evaluate projections 2o (a) such that ||||| S 6
3: Estimate [0,, (#x)) (ef. (7)) and then perform clipping such that (e S Φ
4: Estimate Ok and 0k+1 with adaptive moment estimation method using (8)
f- k-
Until Convergence
6:
Return: 0z
[42, 21]. The primary reason for failure is due to the light-tailed nature of Gaussian distribution which restricts the policy to take action closer to its mean value and thereby restricts exploration. Typical methods to deal with exploration includes adding intrinsic curiosity [45] or entropy based exploration [17] which either requires learning the global dynamics model or estimation of the oc- cupancy density function which extremely expensive for continuous control robotics problem.
In this work, we leverage an alternate approach motivated by the latest research by [42, 21] and focus on developing heavy-tailed parametrization based policy for outdoor navigation problems with additional robotics-driven constraints as detailed in Algorithm 1. We parameterize the policy by heavy-tailed Cauchy distribution given by
To (a(a))) = aτ + ((a ((s))(())))))))))) 0)
(6)
where a is the fixed variance. Now, finally we write the stochastic policy gradient [20] as
Th
[J(Ok,Eu(OL)) - Σ 7122(();) t=
Σ v log To, (a (a.11x)).
(7)
where V.J(0,, (0xk)) denotes the unbiased estimator of gradient VJ((k) at 6z. and Th
Geom(1 - -2/2). Although heavy-tailed parametrization induces exploration which has proven to be extremely beneficial in sparse reward settings [42, 21J, it also induces instability in behaviour due to the high probability of taking extreme action even near optimal regions. Hence, [42] pro- posed gradient tracking to mitigate the above issue. However, instead of momentum tracking with mirror ascent type update as proposed in [42], we use an adaptive moment estimation based method as optimizer with gradient clipping to stabilize the instability induced owing the heavy-tailed parametrization which improves the time taken by gradient tracking based methods.
my = 31mm + (1 - 31)gk.
(8)
2k = 320k + (1 32)
(9)
B = 0x + Uh + mk:
(10)
where my, Uk are the first and second moments respectively and B, 32, c are hyperparamters for the optimizer and gk (J((,, Ex (0x)) with the added constraint of clipping given as J[ & (ox) I á. Another important challenge is to incorporate practical constraints into the learning system which is specific to roboties based problems where the action bounded. To solve the same, we use a simple projection Po)(.) based technique where we project the output from the policy To (a(as) into infinite norm ball which constraints the action space within the bound and can be shown as a ~ To (a(s)) such that ||alleo < 8.
4 Experiments
We conduct detailed analysis and performance comparison of our proposed method against three state-of-the--rr stochastic policy gradient algorithms: TRPO [24], PPO [23], and Reinforce [22]. We used identical policy distribution parameters (e.g. distribution standard deviation a)) and neural net- work architectures with all four algorithms during training and evaluation. The heavy-tailed policies
5

---

ITTONACAnD)
6000
6000
TENO
4000
TEPO
4000
TRaO
PROVCRUCT))
PRojonuts)
Aug. Cumulative
-2000
-4000
-4002
100 -6000
-000
20
Episodes
Episodes
100
120
140 20
Episodes
60
80
100
(b)
Figure 2: Performance comparison of HTRON with Reinforce [22], PPO [23]. TRPO [24], PPO (with Cauchy) and TRPO (with Cauchy) three scenarios: (a) Goal-reaching Baseline, (b) Obstacle avoidance, and (c) Navigating on uneven terrains. Each episode contains 300 times steps (i.c. 30k-42k overall time steps). All the policies are trained with fixed standard deviation a = 0.25. We note that HTRON achieves the highest average reward returns. The plots averaged over 6 random seeds and shows mean and confidence intervals.
6000
IITONNDurs)
6000
INTRONIOuSi
6000
ATROBACunt
Hos snororpteiiii
2000
4000
Cumulative
2000
2000
-2000
-0000
-2000
NW
-0000
-6000
20
100
Episodes
Episodes
Episodes
(a)
(b)
(c)
Figure 3: Performance comparison of HTRON with Reinforce [22], PPO [23], and TRPO [24] policies with large standard deviations (a = 0.5) in three scenarios: (a) Goal-reaching Bascline, (b) Obstacle avoidance, and (c) Navigating vergence by simply increasing standard deviation the policy distribution. Instead, Gaussian policies demonstrate higher instability during training.
terrains. We observe that the Gaussian policies cannot achieve faster policy con-
are implemented and trained with Pytorch using Unity based realistic outdoor simulator, Clearpath Husky robot model, and ROS Melodic platform. The Unity simulator includes diverse terrains and elevations for training and testing. Further, we deploy HTRON on a real Clearpath Husky robot on outdoor terrains.
4.1 Policy Learning Performance
We evaluate the performance of algorithms through the policy convergence rate (see Figure 2). In the goal-reaching bascline scenario, observed that standard Gaussian policy-based algorithms cannot achieve the highest reward return when we incorporate the sparse rewards. In contrast, our heavy-tailed policy is able to obtain maximum rewards in sample-efficient manner as shown in Figure 2(a). This is primarily due to the improved exploratory behavior achieved by the heavy- tailed policy in comparison the standard Gaussian policies.
In Scenario with static obstacles, our experiments showed that the random and sparse obstacle penalties significantly affect policy convergence and stability. In particular, PPO converges to the halfway point towards the goal with multiple collisions. However, our method successfully learns policies to reach the goal within 40 episodes (see Figure 2(b)). This demonstrates the accelerated learning capabilities provided by the heavy-tailed formulation compared to the traditional stochastic policy gradient methods. In Scenario 3, we include I'ttbee penalize the actions that could navigate the robot towards steep elevations. We observe that Reinforce and TRPO based policies struggle to even reach the goal under this setting while PPO manages to reach the goal in only a few episodes without proper convergence.
Further, we performed additional experiments on PPO, TRPO, and Reinforce algorithms by increas-
ing the standard deviation (a 0.5) of their Gaussian policy distribution(see Fig. 3). We observe that even with increasing variance, our algorithm (HTRON) outperforms all the baselines including PPO, TRPO, Reinforce, etc which demonstrates the superiority our algorithm. Additionally, We observe that the Gaussian policies cannot achieve faster or better policy convergence by simply in-

---

Ege-graph
(0)
Figure Trajectories when navigating on different simulated outdoor terrains using HTRON, PPO[23], TRPO[24], Reinforce[22], DWA[30] and Ego-graph[46, 47J: (a) Geal-reaching baseline, (b) Obstacle avoid- ance environment with multiple trees, walls, and unreachable steep elevations. (c) Uneven terrain environment with different levels of elevations and flat terrains. HTRON takes the shortest time steps successfully reach the goal while maintaining the lowest elevation cost on uneven terrains.
HTRON (Dars)
DWA
Ege graph
Figure 5: Navigating real world using HTRON: (a) Goal-reaching scenario, (b) Obstacle avoidance. (c) Uneven terrain navigation. We deploy our algorithm on Clearpath Husky robot demonstrating the case of transferring HTRON on real systems without significant performance degradation. The robot is equipped with VLPI6 LiDAR, and laptop with an Intel i9 CPU and an Nvidia RTX 2080 GPU.
creasing the standard deviation of the policy distribution. Instead, the Gaussian policies demonstrate higher instability during training. Hence, we restricted to a = 0.25 as presented Fig. during our experiments.
4.2 Navigation Performance
We compare our method's navigation performance qualitatively in Figure 4 and quantitatively in Ta- ble 1. The quantitative evaluations of the navigation trajectories are based on the following metrics:
Success Rate: The percentage of successful goal reaching attempts without any collisions or flip-overs out of the total experiments.
Avg. Trajectory Length: The average number of time steps taken by the robot to reach the goal.
Elevation Cost: Norm of the elevation gradient experienced by the robot throughout trajectory (i.e. |/, where 2, is the vector that includes vertical motions of the robot's along a trajectory).
We would like to note that in this work rather than proposing competitive navigation algorithm, the primary objective to highlight the importance of heavy-tailed distributions for faster policy convergence under practical sparse reward settings. However, in addition comparing against the DRL-based methods, we evaluate the navigation performance of two classical algorithms: Dynamic Window Approach(DWA)][3)
and Ego- ego-graph(47]) under the same test conditions. We observe that DWA and Ego-graph outperform all the DRL based methods in terms of success rate during goal reaching and obstacle avoidance. However, HTRON demonstrates comparable better perfor- mance during uneven terrain navigation in terms of all the evaluation metrics.
We further notice that method maintains the highest success rate in all three scenarios while other DRL policies demonstrate a significantly low performance in successful goal-reaching. This is primarily due to the poor convergence in Gaussian policies under sparse reward settings. Further, HTRON reaches the goal faster using fewer time steps than the other three methods. This indicates that our policy has optimized to collect more goal completion rewards by reaching the goal location quickly. Policy convergence plots presented in Figure 2 further validates this argument.
Navigation scenarios and 2 are common in both indoor and outdoor environments. Hence, we further investigate the importance of our algorithm to handle uneven terrains encountered complex outdoor environments. We observe that our method is capable of avoiding steep elevations from the
7

---

Scenario
Algorithm
Success Rafe
Avg. Trajectory Length (# time steps)
Elevation Cost
(%)
(m)
Goal Reaching Baseline
TRPO[24]
32.28
2336
0.764
PPO[23]
45.76
198
0.648
Reinforce[22]
36.91
207
0.655
DWA[30]
100
172
0.645
Ego-gruph(47]
88.86
186
0.637
HTRON(OUrS)
72.46
168
0.641
Obstacle Avoidance
TRPO[24]
18.74
243
0.637
PPO[23]
37.56
224
0.652
Reinforce|22]
8,93
286
0.624
DWA[30]
100
218
0.645
Ego-graph[47]
58.79
238
0.642
HTRON(Ours)
62.21
200
0.639
Navigating on Uneven Terrains
TRPO[24]
25.87
273
1.842
PPO[23]
37.46
264
1.674
Reinforce[22]
18.33
270
1,336
DWA[30]
27.38
268
1.943
Egto-grph(447]
65,89
198
1.316
HTRON((Orr)
71.87
244
1.257
Navigation Comparisons: HTRON outperforms the other DRL methods in terms of success rate and the average time to reach the goal. It also maintains the lowest elevation cost when navigating uneven terrains. Reinforce and TRPO obtain slightly lower elevation costs during goal-reaching and obstacle avoidance scenarios by rotating around flat regions without reaching the actual goal.
Table I:
changing roll and pitch angles before leading to robot flip-overs (see Figure 2(c) and 4(c) ). Hence, trajectories generated by our algorithm significantly minimize the elevation cost experienced by the robot while other methods navigate along steep terrains with higher elevation costs.
algorithm into real Clearpath Husky robot demonstrate navigation capabilities in real outdoor settings. We observe that our method can successfully perform the navigation tasks we trained in the simulator. Sample navigation trajectories from three real outdoor scenarios are presented in Figure 5.
Finally, integrate
Limitations and Analysis
We diseuss the limitations of our method that we expect to address in the future. The robot cannot identify the uneven terrains without experiencing the uneven region due the orientation based elevation sensing. A 3D LiDAR pointcloud based elevation map could be utilized in state space identify the elevation changes in the robot's vicinity without explicitly experiencing elevation changes from the orientation data. Moreover, the heavy-tailedness of the policy distribution could lead convergence instabilities. Especially, behavior when we try to use PPO and TRPO with heavy-tailed distributions. Hence, further inves- tigations need to be conducted develop robust and stable heavy-tailed policy models for higher dimensional state spaces and more complex sparse reward settings.
observe considerable instability and converggggg
6 Conclusion
In this work, we presented a novel approach for adapting heavy-tailed policy based control strategy for efficient outdoor mobile robot navigation HTRON. The proposed method is able overcome the major issue of sparse rewards inherent to DRL based approaches. We show that without hand- crafted reward shaping methods we able accelerate learning capabilities and produce reliable navigation policy. We have assessed the performance of navigation policies in three different sce- narios based on their complexity and corresponding objectives. All policies have been trained in ROS-based
Unity simulator. We report a performance comparison of our method against REINFORCE, PPO, and TRPO algorithms in terms of learning rate and navigation quality. We also deployed HTRON on real Clearpath Husky in outdoor terrains. And finally, we report the limitations, possible improvements, and future work that this approach can lead to robust outdoor navigation.

---

Acknowledgments
This work was supported in part by ARO Grants W91 W911NF191006,, 19F22100002,, and Army Cooperative Agreement W911NF2120076. We acknowledge the support of the Maryland Robotics Center.
References
[1] D. Lee, G. Kang, B. Kim, and D. H. Shim. Assistive delivery robot application for real-world postal services. IEEE Access, 9:141981-141998, 2021. doi: 0o:::0..00//ACCCSS2220.....
[2] Y. Liu and G. Nejat. Multirobot cooperative learning for semiautonomous control in urban search and rescue applications. Journal of Field Robotics, 33(4):512 2016.
[3] A. Thoesen and H. Marvi. Planetary surface mobility and exploration: A review. Current Robotics Reports, 2(3):239-249, 2021.
[4] V. Dhiman, S. Banerjee, B. Griffin, J. M. Siskind, and J. J. Corso. A critical investigation of deep reinforcement learning for navigation. arXiv preprint arXiv:/802.022774 2018.
[5] Y. F. Chen, M. Liu, M. Everett, and J. P. How. Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 285-292. IEEE, 2017.
[6] K. Zhu and T. Zhang. Deep reinforcement learning based mobile robot navigation: A review. Tsinghua Science and Technology, 26(3):77--691, 1 2021.
[7] H. Li, Q. Zhang, and D. Zhao. Deep reinforcement learning-based automatic exploration for navigation in unknown environment. IEEE transactions on neural networks and learning systems, (166):0044-000 2076, 2019.
[8] J. Liang, K. Weerakoon, T. Guan, N. Karapetyan, and D. Manocha. Adaptiveon: Adaptive outdoor navigation method for stable and reliable motions. arXiv preprint arXiv:2205.03517, 2022.
[9] P. Long, T. Fan, X. Liao, W. Liu, H. Zhang, and J. Pan. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 6252-6259. IEEE, 2018
[10] A. Sathyamoorthy, J. Liang, U. Patel, T. Guan, R. Chandra, and D. Manocha. Densecavoid: Real-time navigation in dense crowds using anticipatory behaviors. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 11345-11352. IEEE, 2020.
[11] A. Zai and B. Brown. Deep reinforcement learning action. Manning Publications, 2020.
[12] N. Jiang, S. Jin, and C. Zhang- Hierarchical automatic curriculum learning: Converting sparse reward navigation task into dense reward. Neurocomputiing 360:265-276 20119.
[13] C. Wang, J. Wang, J. Wang, and X. Zhang. Deep-eeinforcmmmet-eemmniii based autonomous uav navigation with sparse rewards. IEEE Internet of Things Journal, 7(7):6180-619) 2020.
[14] H.-T. L. Chiang, A. Faust, M. Fiser, and A. Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters, 4(2):2007-2014 2019.
[15] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.088) 2017.
[16] H. Karnan, G. Warnell, X. Xiao, and Stone. Voila: Visual-observation-nii imitation learn- ing for autonomous navigation. arXiv preprint arXiv:2105.0937/ 2021.
[17] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.

---

[18] A. Konar, B. H. Baghi, and G. Dudek. Learning goal conditioned socially compliant navigation from demonstration using risk-based features. IEEE Robotics and Automation Letters, 6(2): 651-658, 2021.
[19] J. Hare. Dealing with sparse rewards in reinforcement learning. WwXiv:1900009280 2019.
arXiv preprint
[20] A. S. Bedi, A. Parayil, J. Zhang, M. Wang, and A. Koppel. On the sample complex- ity and metastability of heavy-tailed policy search in continuous control. arXiv preprint arXiv:2106.0841/ 2021.
[21] S. Chakraborty, A. Bedi, A. Koppel, Tokekar, and D. Manocha. Dealing with sparse rewards in continuous control robotics via heavy-tailed policies. 2022.
[22] R. J. Williams. Simple statistical fradient- algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229-23 22,,, 1992.
[23] Schulman, Wolski, P. D.hariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170717634770777 2017.
[24] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[25] A. Sathyamoorthy, K. Weerakoon, T. Guan, J. Liang, and D. Manocha. Terrapn: Unstructured terrain navigation through online self-supervised learning. arXiv preprint arXiv:2202. 12873, 2022.
[26] K. Weerakoon, A. J. Sathyamorrthy, U. Patel, and D. Manocha. Terp: Reliable planning
in uneven outdoor environments using deep reinforcement learning. In 2022 International Conference on Robotics and Automation (ICRA), pages 9447-9453, 2022.
[27] B. Patle, A. Pandey, D. Parhi, A. Jagadeesh, et al. A review: On path planning strategies for navigation of mobile robot. Defence Technology, 15(4):582-606 2019.
[28] X. Xiao, B. Liu, G. Warnell, and P. Stone. Motion planning and control for mobile robot navigation using machine learning: survey. Autonomous Robots, pages 1-29, 2022.
[29] E. W. Dijkstra et al. A note on two problems in connexion with graphs. Numerische mathe- matik, 1(I):269-271, 1959.
[30] D. Fox, W. Burgard, and S. Thrun. The dynamic window approach to collision avoidance. IEEE Robotics Automation Magazine, 4(1):23-33, 33, 1997.
[31] Kuffner and M. LaValle. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 2, pages 995-1001. IEEE, 2000.
[32] I.. F. Kavraki, P. Svestka, J.-C. Latombe, and M. H. Overmars. Probabilistic roadmaps for path planning high-dimensional configuration spaces. IEEE transactions on Robotics and Automation, 12(4):566-58)) 1996.
[33] C. Zhou, B. Huang, and P. Fränti. A review of motion planning algorithms for intelligent robots. Journal of Intelligent Manufacturing, pages 1-38, 2021.
[34] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and Levine. Ving: Learning open-world navigation with visual goals. In 2021 IEEE International Conference Robotics and Automa- tion (ICRA), pages 13215-13222. IEEE, 2021.
[35] K. Zhang, Niroui, M. Ficocelli, and G. Nejat. Robot navigation of environments with un- known rough terrain using deep reinforcement learning. In 2018 IEEE International Sympo- sium on Safety, Security, and Rescue Robotics (SSRR), pages 1-7. IEEE, 2018
[36] S. Josef and A. Degani. Deep reinforcement learning for safe local planning a ground vehicle in unknown rough terrain. IEFE Robotics and Automation Letters, 5(4):6748-677) 2020.
10

---

[37] T. Guan, D. Kothandaraman, R. Chandra, A. J. Sathyamorrthy, K. Weerakoon, and D. Manocha. Ga-nav: Efficient terrain segmentation for robot navigation unstructured outdoor environments. IEFF Robotics and Automation Letters, ((3):81388814) 2022. doi:
R0.1109/.RA.2222318822
[38] A. Hussein, E. Elyan, M. M. Gaber, and C. Jayne. Deep reward shaping from demonstrations.
In 2017 International Joint Conference on Neural Networks (IJCNN) pages 510-517. IEEE, 2017.
[39] N. Botteghi, B. Sirmacek, K. A. Mustafa, M. Poel, and S. Stramigioli. On reward shaping for mobile robot navigation: A reinforcement learning and slam based approach. arXiv preprint arXiv:2002.044 2et.2000200444 2020.
[40] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. 2018 IEEE international conference on robotics and automation (ICRA), pages 2292-6299. IEEE, 2018.
[41] H. Ravichandar, A. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learn- ing from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, (ARTICLE):297-330, 2020.
[42] A. S. Bedi, S. Chakraborty, A. Parayil, B. M. Sadler, P. Tokekar, and A. Koppel. On the hidden biases of policy mirror ascent in continuous action spaces. CoRR, abs/2201.12334 2022. URL https:/ /aaxiiv.///ivv.///// org/ abs/22011 12332.
[43] P.-W. Chou. The beta policy for continuous control reinforcement learning. PhD thesis, Mas- ter's thesis. Pittsburgh: Carnegie Mellon University, 2017.
[44] X.-H. Chen, S. Jiang, F. Xu, Z. Zhang, and Y. Yu. Cross-modal domain adaptation for cost- efficient visual reinforcement learning. Advances in Neural Information Processing Systems, 34:12520-12532, 2021.
[45] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
[46] A. Lacaze, Y. Moscovitz, N. DeClaris, and K. Murphy, Path planning for autonomous vehicles driving over rough terrain. In Proceedings of the 1998 IEEE International Symposium on Intelligent Control (ISIC) held jointly with IEEE International Symposium Computational Intelligence in Robotics and Automation ( (CIRA) Intell, pages 50-55, 1998. doi:10.1109/ISIC 1998.71363.
[47] S. Josef and Degani. Deep reinforcement learning for safe local planning of ground vehicle in unknown rough terrain. IEEE Robotics and Automation Letters, 5(4):6748-6755, 2020. doi: 10. 2.0000000..2000001199999


Reviews:
[{"review_id": "zFh3JhBvSb", "reviewer": "Reviewer_cxNk", "summary": "This paper proposes a better policy parameterization that works better with sparse rewards in outdoor navigation tasks. The challenge of training RL agents in navigation tasks is reward sparsity since the rewards are only emitted upon the agent reaches the destination. Prior works use reward shaping/engineering to make training easier, but it requires intensive human efforts. The sparse reward function is easy to design but challenging to RL algorithms. The author believes sparse rewards make exploration difficult, and Gaussian policy used in policy gradient algorithms tends to over-commit to the mean actions. Thus, the author proposes to replace the Gaussian policy with a heavy-tailed distribution that has a higher chance of sampling actions away from the mean. The experimental results in both simulator and real-world show good performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n## Strength\n- The idea of using heavily tailed distribution to solve sparse rewards is sensible and elegant.\n\n## Weakness\n- Lack of experiments: If the argument is that sparse reward is preferred and HITRON can learn a good policy with sparse rewards, it would be necessary to compare PPO/TRPO/Reinforce with intensively tuned rewards. Otherwise, the reason of preferring HITRON over other baselines+tuned reawrds can be weak.\n- Lack of comparison: If the primary benefit of HITRON is better exploration, I'm curious whether PPO/TRPO/Reinforce can match HITRON by increasing the standard deviation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend weak acceptance since this paper has a reasonable motivation and method, and the experiment supports the claim, despite lacking comprehensiveness. Overall, there are no obvious technical flaws. This makes this paper pass the bar of acceptance.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "\n1. Line 182: what is gradient tracking? it would be better to briefly summarize how [42] do this.\n2. \"training instability\" is used a lot in this paper but never formally defined.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "QMiCiXLKS-n", "reviewer": "Reviewer_B9Ro", "summary": "This paper investigates learning three navigation tasks with a husky robot, in simulation and with demonstrations on a real platform. Different from typical methods, this work considers the challenges standard RL algorithms have with sparse rewards, proposing a heavy-tailed approach to improve exploration. To improve learning stability, gradient tracking and clipping was required. The results indicate an increase in success rate for navigating to a goal compared with several baseline algorithms.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper was well written, presented clearly, and easy to follow.\n\nThe primary concern is that the contribution is minor. The paper focuses heavily on the three elected navigation tasks, however, these seem relatively simple problems for reinforcement learning to solve if the imposed sparse reward limitation was lifted, particularly as the ground-truth goal information (non-sparse, distance and angle to the goal) is provided as input to the policy. \n\n It seems a constant variance is used throughout for sampling stochastic actions. Does the choice of variance affect learning, and what effect would having variance as a learnable parameter have on performance? This is particularly interesting for the comparison between Cauchy and Gaussian distributions for exploration. The resulting policies produce non-smooth trajectories, suggesting that more improvements are needed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "One way to make this method more appealing would be to show quantitative learning improvements with a sparse reward over a shaped reward, i.e. show that this task benefits from a sparse reward, and that this method improves learning with such a reward. For now, the reader needs to imagine that a continuous/shaped reward is difficult to design, which is not the case for the presented problem. For this, a different task setup may need to be investigated. \n\nGiven the outcome of this paper is improved exploration for learning with sparse rewards, it is surprising that the Soft Actor-Critic algorithm is missing, from the comparisons and from the extensive reference list. Inclusion, or a discussion on reasons for exclusion, would be useful.\n\nAn ablation of some of the techniques and design choices, for example, the effect of gradient tracking and clipping, and choice of variance, would be useful for the community to understand what is difficult about using heavy-tailed policies.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "\nSpelling / Grammar / Typos:\n“Number of deterministic” - missing “A”\n\n“For instance, majority of the research” \n\n“restricts the policy to take action very”\n\n“for outdoor navigation problem”\n\n“which constraints the action space”\n\n“evaluation.The”\n\n“in outdoor terrains” - on outdoor terrains\n\n“via utilizing policy”\n\n“assessed the performance navigation policies”\n\n“∥∇J(θ k , ξ k (θ k ))∞ ≤ ϕ” - Missing closing double pipes before infinity\n\t\n“motions of the robot’s along a trajectory”\n\n“deployed HTRON on real Clearpath Husky”\n\n“However, instead of momentum tracking with mirror ascent type update as proposed in [42], we use an adaptive moment estimation based method as an optimizer with gradient clipping to stabilize the instability induced owing to the heavy-tailed parametrization which improves the time taken by gradient tracking based methods.” - Difficult sentence, missing hyphens and probably a comma or two.\n"}}, {"review_id": "D9Kz-R3GsjJ", "reviewer": "Reviewer_skdZ", "summary": "Given prior work establishing that heavy-tailed policies help with exploration for sparse reward RL both in theory [20] and practice [21], this work explores evaluating heavy-tailed policies for sparse reward outdoor navigation. Since heavy tailed policies have already been experimentally studied for sparse reward RL in [21], the primary contribution of this paper appears to be the application of this same idea to outdoor navigation problems. The main differences of HTRON from the approach presented in [21] appear to be (1) the use of an adaptive moment estimation method to stabilize policy updates instead of the moment tracking approach used in [21] and (2) an extra projection step to handle bounded action spaces.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) The setting of outdoor navigation is definitely an important one in robotics as a whole, and is a setting where sparse reward RL is very well motivated.\n\n(2) HTRON is thoroughly evaluated on a suite of simulation tasks and a physical outdoor navigation task.\n\nWeaknesses:\n\n(1) The methodological contribution of this paper is not clear to me. The only differences of HTRON from the approach presented in [21] appear to be (1) the use of an adaptive moment estimation method to stabilize policy updates instead of the moment tracking approach used in [21] and (2) an extra projection step to handle bounded action spaces. If the claim is that HTRON is an heavy-tailed RL algorithm specifically designed for navigation problems, then I would expect more structure specific to outdoor navigation to be built into HTRON. Without this, it is hard to understand what HTRON contributes beyond [21] since the impact of the adaptive moment estimation method is not evaluated in experiments and the handling of bounded action spaces is a very minor addition. Indeed, it is strange that no direct comparison between [21] and HTRON is performed given how heavily HTRON is based on [21].\n\n(2) The experimental comparisons are also a bit unclear. From my understanding, the only difference between Reinforce and HTRON in Figure 2 is the use of a heavy-tailed policy in HTRON. If this is indeed correct, then the comparison to PPO and TRPO, which presumably use Gaussian policies is a bit strange. It would make more sense to compare PPO and TRPO to their equivalents if a heavy-tailed policy were used.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The methodological contribution of this paper seems unclear/incremental with respect to prior work on heavy-tailed policies for RL. While applying these prior works to navigation problems is interesting, this alone does not seem sufficient for publication at CoRL unless further evidence is provided (both written and empirical) to differentiate the proposed approach from prior algorithms.\n\nPost Rebuttal:\n\nI appreciate the authors' point though that one core contribution of the work is in the \"design and analysis of the Sparse reward scenarios which are practical and relevant for Outdoor Navigation problems\", which I missed in my initial review. The added comparison to classical algorithms though significantly improves my view of the thoroughness of the empirical evaluation, so I will raise my score to a Weak Accept once I am able to adjust my score. However, I still believe that comparison to [21] should be performed if possible.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The core issue with this paper is it not clear to me what the contribution is relative to [21]. If the contribution is a heavy-tailed RL algorithm specifically designed for navigation problems, then I would expect a lot more structure specific to navigation to be built into HTRON. If the contribution is instead a generally more effective heavy-tailed RL algorithm, then the authors must argue what the benefits of HTRON are relative to [21] and illustrate these benefits through experimental comparison."}}, {"review_id": "ADjcAx_j2bV", "reviewer": "Reviewer_N3Ka", "summary": "The authors propose HTRON, a deep reinforcement learning (DRL) framework intended to support outdoor robot navigation. Three problems of interest (goal-reaching, obstacle avoidance, navigating on uneven terrain) are configured to provide sparse rewards, and heavy-tailed policy parameterizations are utilized to induce exploration. HTRON repeatedly selects a control action for an unmanned ground vehicle (UGV) based on inputs that include distance and relative heading to goal, the robot’s attitude and previous control action, and the contents of a 360-degree laser scan. Compared with other DRL frameworks, HTRON is shown to converge more successfully to an effective navigation policy, and achieves superior success rates and navigation performance over the three representative tasks selected. Although trained in simulation, HTRON is also deployed successfully on a real UGV. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper addresses a topic relevant to robot learning, proposing a novel method for performing outdoor UGV navigation using deep reinforcement learning. \n\n+ The paper is well-organized and clearly written, and the supplemental video provides very helpful illustrative examples of HTRON’s performance in simulation (alongside competing DRL algorithms) and aboard a real UGV. \n\n+ HTRON is shown to address the three selected UGV navigation problems with superior rates of success relative to the selected DRL baseline algorithms, and offers superior navigation performance in nearly all instances as well. \n\n\n- The prospect for high-performance outdoor UGV navigation is severely limited by the decision to work with a single scan line, rather than to fully utilize the 3D volumetric scanning lidar that the authors’ UGV appears to be equipped with. This sensor can be utilized without DRL to successfully achieve each of the three navigation tasks examined in this paper - none of these tasks represent unsolved problems that require the use of DRL. \n\n- The paper would be strengthened by providing clarification on the extent to which the outdoor navigation scenarios encountered in the training process differed (or did not differ) from the simulated scenarios encountered at test-time. \n\n- The paper would also be strengthened by discussing the various ways that classical algorithms (which do not invoke machine learning, and do not require a prior map of the terrain) are capable of solving the outdoor navigation problems discussed in this paper, and possibly also including them in the experimental comparison to motivate the use of DRL to solve these problems. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In HTRON, the authors have contributed a novel deep reinforcement learning framework for solving three outdoor navigation problems for lidar-equipped UGVs: goal-reaching, obstacle avoidance, navigating on uneven terrain. Impressively, the navigation policies learned in simulation outperform other DRL algorithms, and also transfer successfully to real robot hardware. However, the paper would be strengthened by providing stronger motivation and evidence for why DRL is needed to solve these problems. \n\nPost-Rebuttal Comments: The authors' revisions and clarification are much appreciated, and the revisions have strengthened the paper. Taking these into account (especially the new results that have been added to the study) along with the CoRL review criteria, I have decided to elevate my rating to weak accept. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Please clarify how the outdoor navigation scenarios encountered in the training process differed (or did not differ) from the simulated scenarios encountered at test-time.\n\n- Please also clarify the motivations for using DRL to solve the outdoor navigation problems explored in this paper, why only a single scan-line of the lidar is used (which also appears to be down-sampled from the available resolution of the sensor), and why no classical algorithms are included in the experimental comparison. The full 3D field of view of the lidar used in your experiments is likely capable of real-time terrain traversability mapping and path planning that could solve the problems of interest with comparable or superior performance to DRL. "}}]

Author rebuttal:

Decision: Accept (poster)