Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes to use four directions (top, down, left, right) as the general subgoals to learn the high-level policy in a hierarchical reinforcement learning (HRL) setting. Since the directions are task-agnostic and agent-agnostic for the typical navigation jobs, the proposed framework could be more generalizable. Further, to overcome the mismatch problem while training an HRL policy, the authors (1) trained the high-level and low-level policies separately and (2) introduced a slight noise controlled by \epsilon into the perfect low-level controller in the environment. The experimental results in MuJoCo suggest that the proposed framework outperforms various baselines.
strength_and_weaknesses: +) The studied problem is essential, and the proposed idea is reasonable. Using four directions as subgoals for navigation tasks makes more sense than using absolute coordinates in prior works. The results also suggest that the learnable high-level policy is reusable for different low-level controllers (different agents).


-) The manuscript has not been well written. Several typos, grammar errors, duplicated words, or redundant spaces exist. In addition, some notations are misleading. Taking “x+, x-, y+, y-“ as an example, are the x and y variables? Or do they only indicate direction? All of them make the paper hard to follow.


-) Some algorithms proposed in prior works use collected on-policy rollouts to compute empirically expected policy gradients to perform the policy optimization, such as SAC, PPO, and TRPO. Is there a specific reason for using A2C in the experiments? Since the authors claimed the expected A2C algorithm is one of the contributions, a comparison to the mentioned policy gradient algorithm is necessary.


-) There are many missing details in the experiment section. For example, what is the action space for different agents (e.g., the low-level controllers for different agents?)? What are the learning rates to train the high-level policy and the low-level controller? What optimization algorithm (e.g., SGD?) is used in the training stage?
clarity,_quality,_novelty_and_reproducibility: The paper needs to be more carefully polished. The writing quality is concerned. Moreover, the novelty is also a concern since the main contribution is only the discrete subgoals for better generalization. Because there are many missing details in the experiment section mentioned above, I have no confidence in the reproducibility.
summary_of_the_review: The work proposed a simple but intuitive way to discretize subgoal space to improve the generalization ability of an HRL policy. The experimental results show the effectiveness of the proposed framework. However, since the paper is not well polished, it is hard to read and understand the keys under the hood. In addition, because there are many missing details in the experiment section, I have no confidence in the reproducibility. Finally, the usefulness of one of the claimed contributions about “expected policy gradients by A2C” is not validated, which makes the technical contributions weak.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a framework to mitigate a transition mismatch problem in hierarchical reinforcement learning, which utilizes abstract, task-agnostic subgoal generation. Under a certain condition on invariance in subgoals, this work proves thae the error between the hierachical RL and the original full RL can be bounded. The efficacy of the proposed method has been demonstrated on Ant Maze problem in Mujoco, compared against several baselines. 
strength_and_weaknesses: Strengths: 
- This paper is clearly written and easy to follow. The motivation is clearly indicated and the methodlogy is described clearly. 
- The issue of handling mismatch in hierarchical decision settings is a timely and important. 
- The proposed framework looks like making sense in general. 
- The numerical evaluation is done with reasonable details. 
- It is good to provide a theoretical proof on some properties of the proposed scheme.

Weaknesses:
- The theoretical proof relies on the assumption on the epsilon-invariance. It is not clearly explained whether or not this assumption makes practical senses and/or is a simplying one.
- The generalizability of the proposed scheme is not very clearly explained/defended. The notion of subgoals is claimed to be abstract and generic, but it is not very clear to come up with subgoals other than in a geographical domain. 
clarity,_quality,_novelty_and_reproducibility: - The paper is relatively clearly written, and is well-readable. 
- The proposed method seems to be techincally sound and the evaluation is done in a reaonable fashion. 
- The idea seems novel and unique. 
summary_of_the_review: This paper is handling an important problem in hierarchical reinforcement learning by presenting a reasonable framework. The generalizability of the proposed scheme needs to be better justified.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to address the transition mismatch problem in GCHRL using (1) more readily reusable abstract subgoals, and (2) manually injecting stochasticity into the low-level policy as a form of regularization. Both of these contributions are ultimately proposed to improve generalization on unseen tasks where the same subpolicies and high-level policy are applicable. In addition to these contributions, they also propose multiple additional modifications, including using expected gradient updates, as well as a new form of intrinsic reward to train their abstract subgoals. They provide theoretical analysis showing that the error between their method’s hierarchical policy and the optimal policy can be bounded. Lastly, they compare their method to SOTA hierarchical and non-hierarchical algorithms, and demonstrate that their method outperforms these prior approaches, and achieves significantly better zero-shot generalization performance. 
strength_and_weaknesses: Strengths:

* Both major contributions (i.e. using more general subgoals for maze navigation tasks and injecting stochasticity into low-level policies to regularize/improve robustness of the high-level policy) are well-motivated and appear to be novel for this problem setting.

* All important details are described clearly. 

* The results (especially the generalization results) are impressive and indicate a clear benefit of the proposed approach over baselines in continuous control maze navigation tasks. 

* The discussion of related work appears to be complete.

Weaknesses: 

* The main weakness is that there are a number of contributions presented in this work, and there is no comprehensive ablation study demonstrating to what degree each contributes to the improved performance. It would be very helpful to see an ablation study where the effects of (1) abstract (and discretized) subgoals, (2) injected stochasticity, and (3) expected gradients are controlled. Without this, it is difficult to make sense of the results, and to understand the relative importance of the contributions. It would also be useful to see how well the baselines perform when these tricks are applied.

* There are a number of grammatical errors throughout the document that make some points unclear. I suggest another careful pass through to make sure each sentence is well-structured for clarity. There are a number of examples throughout the document, but here is just one example (with some rough ideas on how to improve the grammar/clarity in brackets); in the last sentence of ‘motivation’ paragraph: “[Thus, a] policy based on these subgoals [may also generalize better across related] tasks.”

* There are some minor formatting issues. E.g. backticks are not used as opening quotes in some instances (e.g. defn 3.1, motivation paragraph), and when multiple works are cited they are not included in the same set of parentheses (they appear to each be different calls to \cite). 

* It is unclear how useful the theoretical analysis is. Instead of bounding the the error between an optimal RL method and the proposed method, it could have been nice to see some theoretical motivation for using stochasticity in the first place. But it is possible I am misinterpreting the results—-I did not carefully read the proofs.

* There does not appear to be any results demonstrating the effects of different levels of injected stochasticity. This would be helpful to see how robust the method is to this hyperparameter. It should also point to an important trade-off.

Questions:

* It is not clear to me why the transition mismatch problem is not mitigated by previous relabeling approaches in the multi-task setting. It is claimed that because “successful trajectories are not identical across different tasks, relabeling sampled trajectories may aggravate the mismatch problem”. Is there evidence of this (I see no citation), or some clearer intuition for why this is the case? An example could also be helpful.

* Which of the implementation tricks are applied to the baseline methods? I.e. between (1) abstract subgoals, (2) expected gradients, and (3) using discretized high-level action space. This is unclear to me from the paper, but I could have missed it somewhere.

* What value of $\sigma$ is used for the results in the paper? It is possible I missed this.
clarity,_quality,_novelty_and_reproducibility: * Besides some grammatical issues and problems with clarity on the sentence-level in some places, the exposition is relatively clear and the contributions/results are easily understood.

* The work is generally high quality, but there are some missing ablations/comparisons that are key to understanding the relative impact of each of the contributions.

* The full method appears to be novel in this setting, but the individual ideas are familiar.

* The paper provides sufficient detail that the results should be possible to reproduce.
summary_of_the_review: * This work is generally well-written and the method is well-motivated, but there are a number of contributions the authors make in this paper, and without the proper ablations, it is difficult to understand the relative importance of each. For instance, without these ablation results, the performance benefits may be merely due to the form of the abstract subgoals provided / corresponding intrinsic rewards, rather than the injected stochasticity, and this would make for a comparatively weaker contribution. These concerns will need to be addressed in order for me to recommend acceptance of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript describes the construction of an Hierarchical Reinforcement Learning (HRL) policy for solving the navigation in a maze task. 
Technically, the paper proposes the epsilon-Invariant HRL to leverage task-agnostic abstract subgoals. This approach tries to mitigate the transition mismatch problem ((Zhang et al., 2022) (Levy et al., 2018)) in HRL. Basically, this problem emerges when we try to learn reusable (or task agnostic) subgoals (using some invariable abstract physical quantities as subgolas, e.g. navigation directions), to later build a generalizable high-level policy. The latter evaluates the subgoals with incorrect rewards caused by poorly trained low-level policies. 
The approach in the paper is simple: consider the transition mismatch as a sort of randomness process, this way the high-level policy is trained to adapt to it.
A simple parallel algorithm (the parallel expected gradient advantage actor-critic (PEG-A2C)) is introduced to calculate the expected gradients of different trajectories and update the parameters synchronously, sampling multiple trajectories.
Maze navigation experiments are performed in the MuJoCo suit for maze-navigation tasks. Multiple baselines are used in the experiments (DroQ - SOTA RL model, HIGL - SOTA HRL model, etc.)

strength_and_weaknesses: Strengths:
+ The paper described an HRL policy learning process that offers generalization capability across different mazes and different robots. This is an interesting contribution of the work. The zero-shot robot generalization experiment (Section 4.4, table 2) opens an interesting research direction.


Weaknesses:

-The experimental evaluation has some important limitations:

a) Not enough previous HRL methods which have been specifically designed for the same problem are considered in the experimental evaluation. The manuscript just includes the use of HIGL model (Kim et al., 2021)., but others should have been considered: e.g. (Li et al., 2021), HRAC (Zhang et al. 2022) .

b) The evaluation metric proposed is not appropriate. Average reward can be useful for some problems, but in maze navigation tasks, the success rate is probably the most informative metric. See how all previous works use it previous papers (HIGL for instance), which will allow a direct comparison with them. So, I encourage the authors to change figure 5 to include this metric (and remove figure 6). The reward that the proposed approach accumulated is considerably higher that the one gathered by the rest of models, so I do not expect any surprise when the metric changes, however, the reader will be able to see how "fast" the new model is in achieving a SR of 100% with respect to the rest of the methods in the literature.

c) I encourage the authors to also include previously used maze shapes in the experimental evaluation ((Li et al., 2021)(Kim et al., 2021))

d) For the ablation study, the inclusion of RAND-H and our-oracle is an appropriate choice. I am intrigued, however, by the performance of Ours-Oracle in the generalization experiment in Table 1. How can the model trained using the oracle for a particular type of maze work well in any other type of maze? This means that the high-level policy learned by the proposed model (without the oracle), is far from being effective in generalization tasks.

-See my comments below about the novelty, which a one of the weaknesses.
clarity,_quality,_novelty_and_reproducibility: -Clarity of the paper can be improved. The motivation of including section 3.1 in the way it has been included, results unclear/confusing. The paper leaves in the appendices many details that are important for the understanding of the model.

Novelty:
- The idea of learning in a hierarchical way using RL to control a robot and how to get out of a maze has been explored before (e.g. LESSON model of (Li et al., 2021)).

- The theoretical motivation of the proposed approach is a simple modification of the work of Zhang et al. (2022). While there a k-step adjacent region of the current state using an adjacency constraint is used/explored during learning the policies, this manuscript assumes that the distribution of the randomness of subgoals is in a little region with little variance. Therefore, the paper simply substitutes the search over a k-step adjacent region of the current state of the agent using an adjacency constraint, by the following random algorithm: when training the high-level policy, the simulator moves the robot with random postures and random positions. Overall, this is a minor simplification, and a detailed comparison with the work by Zhang et al. (2022) should have been included.

-Moreover, I can hardly find fundamental differences between the theoretical development in Zhang et al. (2022) and the one in this manuscript. At the end of the day, considering a HRL and a standard RL this paper uses the same Lemma as in (Zhang et al., 2022).


Some minor comments:
-Figure 5: yellow (DroQ performance) curve can't be seen.
-Punctuate the equations, they are part of the text.

Reproducibility: No code is provided, and not enough details are provided to reproduce the results.
summary_of_the_review: Overall, the paper could be of interest to the rest of the community. The ideas are somewhat incremental. The main problem is in the experimental evaluation, and in the clarity with which the model has been laid out (especially in its comparison with the work of Zhang et al. 2022). For the moment, I am inclined to consider the article below the acceptance threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new combination of techniques for learning a hierarchical policy that will generalize across multiple environments. Specifically, using directional sub-goals over position-based sub-goals, adding noise into the sub-goals to force the high-level policy to learn a robust goal-setting strategy, and gradient averaging from multiple trajectories. The reviewers agree that directional sub-goals enable the agent to learn a more general policy. However, as the reviewers point out, there needs to be more evidence to understand how using randomness impacts the learning process or the agent's ability to generalize across environments. Furthermore, the experiments claim state-of-the-art performance but only use three trials and do not account for the hyperparameter selection process in the results. Addressing these gaps will make the results much more compelling and provide valuable insights into understanding the challenges of learning a policy that generalizes over environments. 
justification_for_why_not_higher_score: There paper does not conduct experiments to accepted standards of practice nor does it provide evidence to how the design decisions enable better generalization. Overall this leads to possible misunderstandings in what is important for learning policies that generalize to multiple environments.
Decision: Reject