Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a better policy parameterization that works better with sparse rewards in outdoor navigation tasks. The challenge of training RL agents in navigation tasks is reward sparsity since the rewards are only emitted upon the agent reaches the destination. Prior works use reward shaping/engineering to make training easier, but it requires intensive human efforts. The sparse reward function is easy to design but challenging to RL algorithms. The author believes sparse rewards make exploration difficult, and Gaussian policy used in policy gradient algorithms tends to over-commit to the mean actions. Thus, the author proposes to replace the Gaussian policy with a heavy-tailed distribution that has a higher chance of sampling actions away from the mean. The experimental results in both simulator and real-world show good performance.
strengths_and_weaknesses: 
## Strength
- The idea of using heavily tailed distribution to solve sparse rewards is sensible and elegant.

## Weakness
- Lack of experiments: If the argument is that sparse reward is preferred and HITRON can learn a good policy with sparse rewards, it would be necessary to compare PPO/TRPO/Reinforce with intensively tuned rewards. Otherwise, the reason of preferring HITRON over other baselines+tuned reawrds can be weak.
- Lack of comparison: If the primary benefit of HITRON is better exploration, I'm curious whether PPO/TRPO/Reinforce can match HITRON by increasing the standard deviation.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend weak acceptance since this paper has a reasonable motivation and method, and the experiment supports the claim, despite lacking comprehensiveness. Overall, there are no obvious technical flaws. This makes this paper pass the bar of acceptance.
issues: 
1. Line 182: what is gradient tracking? it would be better to briefly summarize how [42] do this.
2. "training instability" is used a lot in this paper but never formally defined.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose HTRON, a deep reinforcement learning (DRL) framework intended to support outdoor robot navigation. Three problems of interest (goal-reaching, obstacle avoidance, navigating on uneven terrain) are configured to provide sparse rewards, and heavy-tailed policy parameterizations are utilized to induce exploration. HTRON repeatedly selects a control action for an unmanned ground vehicle (UGV) based on inputs that include distance and relative heading to goal, the robot’s attitude and previous control action, and the contents of a 360-degree laser scan. Compared with other DRL frameworks, HTRON is shown to converge more successfully to an effective navigation policy, and achieves superior success rates and navigation performance over the three representative tasks selected. Although trained in simulation, HTRON is also deployed successfully on a real UGV. 
strengths_and_weaknesses: + The paper addresses a topic relevant to robot learning, proposing a novel method for performing outdoor UGV navigation using deep reinforcement learning. 

+ The paper is well-organized and clearly written, and the supplemental video provides very helpful illustrative examples of HTRON’s performance in simulation (alongside competing DRL algorithms) and aboard a real UGV. 

+ HTRON is shown to address the three selected UGV navigation problems with superior rates of success relative to the selected DRL baseline algorithms, and offers superior navigation performance in nearly all instances as well. 


- The prospect for high-performance outdoor UGV navigation is severely limited by the decision to work with a single scan line, rather than to fully utilize the 3D volumetric scanning lidar that the authors’ UGV appears to be equipped with. This sensor can be utilized without DRL to successfully achieve each of the three navigation tasks examined in this paper - none of these tasks represent unsolved problems that require the use of DRL. 

- The paper would be strengthened by providing clarification on the extent to which the outdoor navigation scenarios encountered in the training process differed (or did not differ) from the simulated scenarios encountered at test-time. 

- The paper would also be strengthened by discussing the various ways that classical algorithms (which do not invoke machine learning, and do not require a prior map of the terrain) are capable of solving the outdoor navigation problems discussed in this paper, and possibly also including them in the experimental comparison to motivate the use of DRL to solve these problems. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: In HTRON, the authors have contributed a novel deep reinforcement learning framework for solving three outdoor navigation problems for lidar-equipped UGVs: goal-reaching, obstacle avoidance, navigating on uneven terrain. Impressively, the navigation policies learned in simulation outperform other DRL algorithms, and also transfer successfully to real robot hardware. However, the paper would be strengthened by providing stronger motivation and evidence for why DRL is needed to solve these problems. 

Post-Rebuttal Comments: The authors' revisions and clarification are much appreciated, and the revisions have strengthened the paper. Taking these into account (especially the new results that have been added to the study) along with the CoRL review criteria, I have decided to elevate my rating to weak accept. 
issues: - Please clarify how the outdoor navigation scenarios encountered in the training process differed (or did not differ) from the simulated scenarios encountered at test-time.

- Please also clarify the motivations for using DRL to solve the outdoor navigation problems explored in this paper, why only a single scan-line of the lidar is used (which also appears to be down-sampled from the available resolution of the sensor), and why no classical algorithms are included in the experimental comparison. The full 3D field of view of the lidar used in your experiments is likely capable of real-time terrain traversability mapping and path planning that could solve the problems of interest with comparable or superior performance to DRL.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Given prior work establishing that heavy-tailed policies help with exploration for sparse reward RL both in theory [20] and practice [21], this work explores evaluating heavy-tailed policies for sparse reward outdoor navigation. Since heavy tailed policies have already been experimentally studied for sparse reward RL in [21], the primary contribution of this paper appears to be the application of this same idea to outdoor navigation problems. The main differences of HTRON from the approach presented in [21] appear to be (1) the use of an adaptive moment estimation method to stabilize policy updates instead of the moment tracking approach used in [21] and (2) an extra projection step to handle bounded action spaces.
strengths_and_weaknesses: Strengths:

(1) The setting of outdoor navigation is definitely an important one in robotics as a whole, and is a setting where sparse reward RL is very well motivated.

(2) HTRON is thoroughly evaluated on a suite of simulation tasks and a physical outdoor navigation task.

Weaknesses:

(1) The methodological contribution of this paper is not clear to me. The only differences of HTRON from the approach presented in [21] appear to be (1) the use of an adaptive moment estimation method to stabilize policy updates instead of the moment tracking approach used in [21] and (2) an extra projection step to handle bounded action spaces. If the claim is that HTRON is an heavy-tailed RL algorithm specifically designed for navigation problems, then I would expect more structure specific to outdoor navigation to be built into HTRON. Without this, it is hard to understand what HTRON contributes beyond [21] since the impact of the adaptive moment estimation method is not evaluated in experiments and the handling of bounded action spaces is a very minor addition. Indeed, it is strange that no direct comparison between [21] and HTRON is performed given how heavily HTRON is based on [21].

(2) The experimental comparisons are also a bit unclear. From my understanding, the only difference between Reinforce and HTRON in Figure 2 is the use of a heavy-tailed policy in HTRON. If this is indeed correct, then the comparison to PPO and TRPO, which presumably use Gaussian policies is a bit strange. It would make more sense to compare PPO and TRPO to their equivalents if a heavy-tailed policy were used.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The methodological contribution of this paper seems unclear/incremental with respect to prior work on heavy-tailed policies for RL. While applying these prior works to navigation problems is interesting, this alone does not seem sufficient for publication at CoRL unless further evidence is provided (both written and empirical) to differentiate the proposed approach from prior algorithms.

Post Rebuttal:

I appreciate the authors' point though that one core contribution of the work is in the "design and analysis of the Sparse reward scenarios which are practical and relevant for Outdoor Navigation problems", which I missed in my initial review. The added comparison to classical algorithms though significantly improves my view of the thoroughness of the empirical evaluation, so I will raise my score to a Weak Accept once I am able to adjust my score. However, I still believe that comparison to [21] should be performed if possible.
issues: The core issue with this paper is it not clear to me what the contribution is relative to [21]. If the contribution is a heavy-tailed RL algorithm specifically designed for navigation problems, then I would expect a lot more structure specific to navigation to be built into HTRON. If the contribution is instead a generally more effective heavy-tailed RL algorithm, then the authors must argue what the benefits of HTRON are relative to [21] and illustrate these benefits through experimental comparison.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper investigates learning three navigation tasks with a husky robot, in simulation and with demonstrations on a real platform. Different from typical methods, this work considers the challenges standard RL algorithms have with sparse rewards, proposing a heavy-tailed approach to improve exploration. To improve learning stability, gradient tracking and clipping was required. The results indicate an increase in success rate for navigating to a goal compared with several baseline algorithms.
strengths_and_weaknesses: This paper was well written, presented clearly, and easy to follow.

The primary concern is that the contribution is minor. The paper focuses heavily on the three elected navigation tasks, however, these seem relatively simple problems for reinforcement learning to solve if the imposed sparse reward limitation was lifted, particularly as the ground-truth goal information (non-sparse, distance and angle to the goal) is provided as input to the policy. 

 It seems a constant variance is used throughout for sampling stochastic actions. Does the choice of variance affect learning, and what effect would having variance as a learnable parameter have on performance? This is particularly interesting for the comparison between Cauchy and Gaussian distributions for exploration. The resulting policies produce non-smooth trajectories, suggesting that more improvements are needed.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: One way to make this method more appealing would be to show quantitative learning improvements with a sparse reward over a shaped reward, i.e. show that this task benefits from a sparse reward, and that this method improves learning with such a reward. For now, the reader needs to imagine that a continuous/shaped reward is difficult to design, which is not the case for the presented problem. For this, a different task setup may need to be investigated. 

Given the outcome of this paper is improved exploration for learning with sparse rewards, it is surprising that the Soft Actor-Critic algorithm is missing, from the comparisons and from the extensive reference list. Inclusion, or a discussion on reasons for exclusion, would be useful.

An ablation of some of the techniques and design choices, for example, the effect of gradient tracking and clipping, and choice of variance, would be useful for the community to understand what is difficult about using heavy-tailed policies.

issues: 
Spelling / Grammar / Typos:
“Number of deterministic” - missing “A”

“For instance, majority of the research” 

“restricts the policy to take action very”

“for outdoor navigation problem”

“which constraints the action space”

“evaluation.The”

“in outdoor terrains” - on outdoor terrains

“via utilizing policy”

“assessed the performance navigation policies”

“∥∇J(θ k , ξ k (θ k ))∞ ≤ ϕ” - Missing closing double pipes before infinity
	
“motions of the robot’s along a trajectory”

“deployed HTRON on real Clearpath Husky”

“However, instead of momentum tracking with mirror ascent type update as proposed in [42], we use an adaptive moment estimation based method as an optimizer with gradient clipping to stabilize the instability induced owing to the heavy-tailed parametrization which improves the time taken by gradient tracking based methods.” - Difficult sentence, missing hyphens and probably a comma or two.
review_title: null

Meta-review: metareview: Summary:
The paper investigates an application of heavy-tailed Deep RL to outdoor navigation tasks.  Three problems (goal reaching, obstacle avoidance, urban navigation) are tackled.  The main thrust of the paper is that heavy-tailed distributions can deal with sparse rewards better than standard distributions, and are more likely to sample a variety of actions to support learning.  Simulation and real world tasks are used to evaluate the approach.  The approach shows good performance in both simulation and reality, and compared to a set of selected baselines.

The paper is moderately original, and some extra experimentation may be needed.  The paper in well-written and understandable in general.  It is likely to represent an incremental change in state of the art and impact may be limited.  The approach is in scope for CoRL and contains sufficient robotics focus.  

Strengths
•	The topic is relevant to the community
•	Results are encouraging, the algorithms performs well
•	The tasks selected as important and relevant
•	The paper is generally well written, carefully orgasnised, and has excellent supplementary material.

Weaknesses
•	The work is similar to some of the previous literature, and the amount of scientific novelty is unclear.  The main concern is that the contribution may be minor.
•	The approach uses only a single scan line, meaning most of the capabilities of the sensor aren’t used
•	Standard SLAM solutions can solve the problems without the use of learning.
•	There is no discussion on the differences between simulated and real test scenarios and how it affects algorithmic performance
•	The work should be placed into the broader context of outdoor navigation literature so the approach can be properly assessed.  This would be best done during literature review, but also by including non-learning algorithms to the baseline comparisons
•	The experimental section could be bolstered by investigating the replacement of heavy-tailed policies in other RL algorithms, in addition to adding ‘classical’ baselines as above.


***Update***
Overall, the authors have been responsive to the reviewers comments and the paper is much stronger as a result.  The reviewers are split between reject and accept, in this case I think there is sufficient interest to the research community to accept as a poster, but the area of research as a whole is worthy of much fuller studies in the future.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)