Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary

This paper proposes a hierarchical reinforcement learning algorithm in an attempt to improve the explainability of RL agents via the goals proposed by the higher level policy.  

Strengths 

The problem of designing safer and more transparent RL agents is an important and rather neglected one, so it is nice to see papers on this topic.  

Weaknesses 

However, I think the paper makes some unsubstantiated claims, lacks thorough empirical evaluations, and is not very well-motivated and situated in the broader RL literature. 

First of all, the proposed approach is only evaluated on two very simple tasks (Mountain Car and Lunar Lander) and it is only compared with two ablations of the proposed method. At the very minimum, the paper should include comparisons with a strong RL baseline (i.e. SAC, PPO etc.), a strong HRL baseline (i.e. feudal networks), and a behavioral cloning baseline (i.e. vanilla BC, Dagger, or GAIL) since the algorithm includes elements of all these. I would be quite surprised if SAC and BC don’t achieve similar or much better performance than the proposed methods on these simple environments. Given that you have access to an expert, it is really unclear to me why you would want to use your method versus simple behavioral cloning. There needs to be better  motivation. 

I also do not think the choice of environments is well-motivated. Besides the fact that they are quite simple for current SOTA, they do not strike me as particularly good for emphasizing explainability or for using a hierarchical agent (since they can be easily solved without HRL). I also don’t understand how the goals are useful in this particular case. Given that these are fixed environments, once you’ve trained a RL policy you can accurately predict the agent’s trajectory including all the visited states rather than only some of them (as the proposed method does). I think a more interesting use case would be to test the method on a new environment and show the goals selected by the higher level policy since a typical RL method wouldn’t be able to predict what states it will visit in a new environment without a simulator or running the policy. I suggest using environments that are better suited for your goals and approach and where other methods might fail or would be hard to explain. 

In section 4.1 you make claims regarding the “resistance to small errors” and to “dynamics” / stochasticity which are not supported by any experiments or theory. I suggest either removing them or backing them up with some empirical evidence. 

Finally, I think the core idea of the paper needs to be better motivated. After reading the paper, I am not fully convinced that the proposed architecture to generate goal-based explanations is very useful for understanding and supervising an agent’s behavior. If you consider the environment in which you are training, then I don’t see how predicting goals is better than simply using a standard RL to predict actions and use the simulator to predict future states. If you consider a new environment, I don’t see how the proposed method can generalize if the state space is different so it won’t know what goals to output or its predictions will be off. 

Also, what is the level of granularity for generating the goals (i.e how often the higher level policy acts relative to the lower level policy) and is there a way to tune this? This seems like an important design choice which is not discussed very much in the paper. 

Recommendation

Given the above points, I do not think the paper is ready for publication. 

review_title: Not enough evidence for the claims 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a hierarchical RL method where the high level controller produces a series of sub-goals in an open-loop fashion which the low-level controller attempts to reach sequentially with the aim of maximising task rewards. The agent is trained using an extension of Hindsight Actor-Critic (HAC) algorithm. The algorithm also leverages a model-free flat policy trained on task rewards as an expert. The approach is evaluated on two tasks: Mountain Car and Lunar Lander. 

The related work section lacks a thorough analysis of existing literature in this domain. Additionally, the lack of baselines is quite concerning, I would have liked to see a comparison with closed loop hierarchical agents, or model based hierarchical agents.

1. While the motivation of the work is interesting, the paper lacks novelty and fails to provide any comparisons with existing HRL methods that could also directly fit in the category of explainable RL methods. There have been several papers attempting sub-goal discovery in HRL which are not discussed in this work. Hence, it is hard to assess the novelty and how it stands in comparison to existing methods. Some recent work in this direction: ([Jurgenson et. al., 2019](https://arxiv.org/abs/1906.05329), [Parascandolo et al., 2019](https://arxiv.org/abs/2004.11410), [Nasiriany et al., 2019](https://arxiv.org/abs/1911.08453))

2. To my understanding, the method only works if the low-level controller has near optimal performance as otherwise setting open-loop goals would not be plausible. For example, the agent might diverge significantly from the path expected by the high level-controller while reaching one of the subgoals and hence the rest of the plan is not relevant anymore. This limits the usefulness of this method significantly, making it only plausible in deterministic settings and it is unlikely that it would generalise outside of the distribution of data seen during training. Could you comment on how this could be addressed?

3. Connected to that, other work on model-based HRL also seems directly relevant in this context where there is no need to assume necessarily that the low-level controller is perfect as the high level transition model is learned and leveraged during planning to find a most rewarding series of subgoals. This has an additional benefit of not assuming the low-level policy is perfect and rely on a learned transition model of behaviour for the agent. It also does not rely on having access to an expert low-level controller to provide demonstrations. It would be nice to better motivate the reasoning behind assuming a perfect low-level controller and discuss its failure cases. Relevant works in this direction include  an ([Nasiriany et al., 2019](https://arxiv.org/abs/1911.08453 ), [Pierrot et al, 2020](https://arxiv.org/abs/2007.13363))

4. It is not clear to me how many goals are queried from the high level controller at the start of an episode, the paper states “...,this process repeats until the desired amount of goals has been collected.” It would be good to discuss this in the main text.

In summary, while the motivation seems sound and important, the problem the paper is addressing is not particularly novel. At the same time, the assumptions made seem quite restrictive to me, the experiments are not particularly convincing and rigorous comparisons to baselines are missing.

Hence unfortunately, I don’t believe this paper, in its current form, is appropriate for publication. But I encourage the authors to improve their related work section, discuss and compare their method with existing approaches to subgoals discovery and model-based HRL to better motivate the novelty and significance of this work.

review_title: Official Blind Review #4

final_score: rating: 3: Clear rejection
review_content: The manuscript proposes (1) HAC-General with Teacher (GT), an extension of Hindsight Actor-Critic (HAC) that incorporates the environment reward as part of the goal formulation, and (2) "goal-based explanations", a framework in which the agent is tasked to produce intermediate goal states.

Good things:

+ The problem setting is important and interesting, and the paper lists a good number of applications where adding interpretability to policies would improve their applications to impactful real-life settings.
+ The idea of extending HAC to goal-less environments is also interesting, as it could potentially shine a method for systematically utilise HRL algorithms on standard RL environments.

Concerns:

1. The paper overall doesn't flow well, which makes it difficult to evaluate the novelty and quality of the method.

2. The concept of "goal-based explanations" is confusing. Most modern HRL work assumes the same problem decomposition, that that the agent needs to learn a goal-proposing policy (which might give additional signals such as rewards, embedding of the state, etc) and a acting policy that proposes environment-compatible actions. Thus it is unclear how this setup differs particularly from others.

3. The paper doesn't do a good job at explaining how looking at a series of goals (which in this case are just states) may be interpreted, and how these would correspond to "explanations". Note that I do not wish to be nitpickinging about this particular naming choice, but "explanations" almost raises the expectation that the end user would see some sort of dense (and possibly language-based) description of the policy.

4. GT builds on HAC, but HAC is never properly introduced. Section 3 can be mostly summarised as informing the reader that (a) HAC requires a hand-tuned goal proposal function, (b) that it doesn't utilise the environment reward (but it is not clarified why this would be a problem), and (c) that it has something to do with the problem of "non-stationarity". So, while section 3.1 does attempt to define then the differences between HAC and GT, it is extremely difficult to understand GT without clarifying how the manuscript intends HAC to look like.

5. The terminology used in Section 3 is extremely confusing. What is a "goal/action" (or "action/goal")? What does it mean for a policy to reach it? What is the difference between a normal action and a "hindsight action"? It would be good if all of these things were properly defined, and not left only to graphical form in figures 2, 3, and 4 (which all look extremely similar, and not that clarifying).

6. Section 3.2 declares that the goal of the work is to "create an explainable agent", and thus it is fair to use a pretrained teacher policy. However, (a) HAC does not require such a training setup, so it's unclear how to fairly assess whether GT is an improvement over it, and (b) it brings the method much closer to imitation learning, which the paper does not review at all. There are in fact methods that distill this sort of knowledge from teachers (e.g. https://arxiv.org/abs/1803.03835, https://arxiv.org/abs/1511.06295 https://arxiv.org/abs/2002.08037), and the proposed method looks extremely similar to them. It is also unclear how such a teacher policy would be trained to begin with, and how GT deals with e.g. probable sub-optimality of this teacher.

7. The experimental setting can be greatly improved. The paper proposes to test GT against HAC on mountain car and lunar lander, but (a) both environments are extremely easy to solve, and (b the difference in performance is probably due to HAC being presented an extremely sparse reward setting.

Due to these concerns, I currently cannot recommend acceptance, however I'd be willing to chance my score if the authors were to improve the manuscript by:

- Providing a detailed explanation of HAC, and how GT fundamentally _improves_ it towards achieving better interpretability.
- Providing a comparison (at least narrative-wise) against modern imitation learning literature.
- Improve the experimental setup by:
  a. At least having another modern RL algorithm as a baseline;
  b. Providing a fairer adjustment to HAC (or at least making an argument on how it is currently fair);
  c. Attempting a less trivial environment.

review_title: It is hard to understand the method and how it compares against existing literature

final_score: rating: 3: Clear rejection
review_content: The paper proposes a hybrid imitation learning/reinforcement learning method for learning hierarchical policies where the top layer provides sub-goals and desired cumulative rewards and the bottom layer learns to meet these goals. The advantage of such a decomposition is interpretability of the learned policy. The algorithm is evaluated on MountainCar and LunarLander from OpenAI’s gym. The authors show that their imitation learning/RL scheme is able to solve both tasks while producing reasonable sub-goals. 

I found the sub-goal for interpretability idea very interesting. However the paper lacks in clarity when presenting the algorithm and in depth when evaluating it. For the presentation, the algorithm box should be commented and its main components should be explained in the main paper. Since most of the learning is carried away by HER (which is not even properly referenced in the algorithm box), a background section should be added. The paper should aim to be more self-contained. The idea of adding the cumulative rewards as a goal is easy to grasp and its explanation can be shortened, leaving space for a more thorough explanation of the actual learning of the hierarchical policy. 

For the experiments part, important details are left out. For instance what is the ratio of imitation learning and reinforcement learning? How much imitation learning is needed for learning the task? How is the black-box policy obtained? What is its performance? To which extent the low level policy actually depends on the goals given by the higher level policy? Since imitation learning is involved for both layers, one could imagine that the top layer is learning to produce reasonable goals, the lower layer is learning to produce reasonable policies but there is only a weak connection between the two, and the lower level is only giving the illusion of tracking the top level’s goals.

I’m also unsure about some of the motivations. It is stated in the introduction that the drawback of black-box policies is that they can have surprising behaviors when facing unexpected states. How is the proposed policy immune to this phenomenon? 

Finally, the number of intermediary goals should be discussed somewhere. In robotics, it is common to define the policy as generating a trajectory in joint space and to let a lower level position controller track the trajectory. One can also easily modify OpenAI’s Mujoco tasks to be position controlled and not torque controlled. Would a policy defined in this MDP, where an action is a desired joint configuration of the next state, be de facto interpretable, or is there some constraint in keeping the number of subgoals small to be human readable? 

Overall, I like the direction and I think the plots showing the produced subgoals are very encouraging. However many details of the algorithm are left out and the experiments are not thorough enough to be fully convincing.

review_title: Many details left out

Meta-review: This paper proposes a method for hierarchical decision making where the intermediate representations between levels of the hierarchy are interpretable. I personally really like this general direction, as did most of the reviewers. Unfortunately, it was felt that, even after discussion, this paper is not ready for publication. To summarize the general spirit of the objection to this paper, all reviewers found that the experimental section was faulty and did not match the claims of the paper. Specifically, criticism here surrounded first, the choice of experimental setting, which was not considered to be the best for testing interpretable hierarchical decision making approaches; and second, the choice of comparison/baselines, which did not give sufficient security that the results produced by the proposed approach were sufficiently impressive.

I am satisfied that the reviewers considered the paper fairly, gave constructive criticism, and took onboard the author feedback. As a result, I am recommending rejection. I nonetheless think that the high quality feedback provided here will enable the authors to prepare follow-up experiments that may show their method in a more robust positive light, and encourage them to submit to a future conference, once armed with such results.
Decision: Reject