Explainable Reinforcement Learning Through Goal-Based Interpretability
Authors: Gregory Bonaert
Youri Coppens
Denis Steckelmacher
Ann Nowe
Venue:  (2021)
Keywords: explainable reinforcement learning
hierarchical reinforcement learning
goal-based interpretability

Abstract: Deep Reinforcement Learning agents achieve state-of-the-art performance in many tasks at the cost of making them black-boxes, hard to interpret and understand, making their use difficult in trusted applications, such as robotics or industrial applications. We introduce goal-based interpretability, where the agent produces goals which show the reason for its current actions (reach the current goal) and future goals indicate its desired future behavior without having to run the environment, a useful property in environments with no simulator. Additionally, in many environments, the goals can be visualized to make them easier to understand for non-experts. To have a goal-producing agent without requiring domain knowledge, we use 2-layer hierarchical agents where the top layer produces goals and the bottom layer attempts to reach those goals. 

Most classical reinforcement learning algorithms cannot be used train  goal-producing hierarchical agents. We introduce a new algorithm to train these more interpretable agents, called HAC-General with Teacher, an extension of the Hindsight Actor-Critic (HAC)  algorithm that adds 2 key improvements: (1) the goals now consist of a state $s$ to be reached and a reward $r$ to be collected, making it possible for the goal-producing policy to incentivize the goal-reaching policy to go through high-reward paths and (2) an expert teacher is leveraged to improve the training of the hierarchical agent, in a process similar but distinct to imitation learning and distillation. Contrarily to HAC, there is no requirement that environments need to provide the desired end state. Additionally, our experiments show that it has better performance and learns faster than HAC, and can solve environments that HAC fails to solve.

Under review as a conference paper at ICLR 2021
EXPLAINABLE REINFORCEMENT THROUGH GOAL-BASED EXPLANATIONS
LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Reinforcement Learning agents achieve state-of-the-art performance in many tasks at the cost of making them black-boxes, hard to interpret and un- derstand, making their use difficult in trusted applications, such robotics or in- dustrial applications. We introduce goal-based interpretability, where the agent produces goals which show the reason for its current actions (reach the current goal) and future goals indicate its desired future behavior without having to run the environment, useful property in environments with no simulator. Addition- ally, in many environments, the goals can be visualised to make them easier to understand for non- non-experts. To have a goal- prodducng agent without requiring do- main knowledge, we use 2-layer hierarchical agents where the top layer produces goals and the bottom layer attempts to reach those goals.
Most classical reinforcement learning algorithms cannot be used train goal- producing hierarchical agents. We introduce new algorithm to train these more interpretable agents, called HAC-General with Teacher, an extension of the Hindsight Actor-Critic (HAC) algorithm (Levy et al., 2019) that adds 2 key im- provements: (1) the goals now consist of state s to be reached and reward r to be collected, making it possible for the goal-producing policy to incentivize the goal-reaching policy to go through high-reward paths and (2) expert teacher is leveraged to improve the training of the hierarchical agent, in a process similar but distinct to imitation learning and distillation. Contrarily to HAC, there is no requirement that environments need to provide the desired end state. Additionally, our experiments show that it has better performance and learns faster than HAC, and can solve environments that HAC fails to solve.
INTRODUCTION
Deep learning has had a huge impact on Reinforcement Learning, making it possible to solve certain problems for the first time, vastly improving performance in many old problems and often exceeding human performance in difficult tasks (Schrittwieser et al., 2019: Badia et al. 2020). These improve- ments come price though: deep agents black-boxes which are difficult to understand and their decisions are hard to explain due to the complexity and non-obvious behavior of neural net- works. In safety-critical applications, it is often fundamental respected or to understand what the behavior of the agent will be (García & Fernández) 20155) Bragg & Habli 2018). Simply observing the behavior of the agent is often not enough, since it might take its actions for the wrong reasons or it might have surprising behavior when faced with an unexpected state. Ideally, the agent would explain its behavior, which would allow for auditing, accountability, and safety-checking (Puiutta & Veith, 2020). unlocking the use of Reinforcement Learning systems in critical areas such as roboties, autonomus driving, or industrial applications.
check that certain properties are
We provide three contributions to make more interpretable deep agents. First, we develop new type of explanation for the agent's behavior. Imagine the following scenario: a robotic agent has to traverse a difficult terrain until it reaches a specific building, where it collects reward. The agent decomposes its task into a series of goals (for example, positions it has to reach) and tries to reach these goals successively until it reaches the reward zone. The agent is more interpretable since it explicitly produces the successive goals it is trying to accomplish: the current goal explains its short- term behavior (the joint movements are done to reach the current goal position) and the remaining

---

Under review as a conference paper at ICLR 2021
goals help understand the agent's overall plan to solve the task and predict its future behavior. We call goal-based explanation or goal-based interpretability the use of plan composed by a series of goals.
Both model-based reinforcement learning (Moerland et al. 2020) and planning techniques (Fox et al. ( appear similar to goal-based explanations but there are important differences that make this technique novel. Goal-based explanations do not require learning a model of the environment (neither the reward function nor the transition function), thus being compatible with both model-free and model-based reinforcement learning. Planning can be a useful explainability technique, but it has few limitations: it typically requires knowing the end goals, they often cannot be applied to complex Markov Decision Problems and they may have difficulty handling very large or continuous action spaces or state spaces. Our approach suffers from none of these limitations.
Second, we develop a method to make the agent produce the goals that add interpretability. To do so, the agent structured as 2-1evel hierarchy of policies, with a goal-picking policy that produces goals and a goal-reaching policy that attempts to reach them. Goals are (state, minimum desired reward) pairs, meaning the goal-reaching policy has to reach a specific state in at most H steps and collect minimum amount of reward along the way. To create a goal-based explanation, the gial-iicking policy is queried repeatedly: given the agent's 's state s. we query for the current goal 91 T1); we then assume the agent reaches the state $1 and query for the next goal Go = (82, r2); this process for a fixed amount of steps per environment, though in future work more sophisticated algorithms to determine the adequate number of goals could be compared.
Our third contribution is developing HAC-General, a new algorithm specifically designed to train goal-producing hierarchical agents. This algorithm builds upon the Hindsight Actor-Critic (HAC) algorithm (Levy et al., 2019) and makes it more widely applicable by not requiring the environment to provide an explicit end-goal. Instead of trying reach the end-goal as fast as possible and ignoring the environment's 's rewards, the HAC-General algorithm trains the agent to maximize the collected reward. Our extension tries to preserve the key property that makes the Hindsight Actor- Critic algorithm effective: having an effective strategy to deal with non-stationarity by giving the illusion that the policies in sub-levels are optimal. The HAC-General algorithm also able to leverage a black-box expert to improve and speed up the training for the hierarchical agent.
BACKGROUND & RELATED WORK
2.1 EXPLAINABLE REINFORCEMENT LEARNING
The Reinforcement Learning community has recognized the need for interpretable and explainable agents, and researchers have developed several methods to add explainability and interpretability. Puiutta & Veith (2020) survey explainability techniques; we briefly describe some key methods.
To add interpretability. saliency-map methods determine the importance of each input feature for the policy when it generates its output. Perturbation-base importance by perturbing different parts of the input and measuring the change in the policy's 's output. The larger the change in output, the more important the feature; the magnitude of the change quanti- fies the relative importance of features, making it possible to build the saliency map. In object-based saliency maps (Iyer et al., 2018), in addition measuring the importance of raw features, they also measure the importance of the whole objects present in the image. The importance of each object is measured by masking it and measuring the change in the policy's s output. Thus, a higher-level object saliency map is created which can be more easily interpreted by non-experts.
methods (Greydanus et al. 2018) measure
Another approach is to distill the policy of the black-box agent into a simpler, more interpretable model while trying to preserve the behavior and performance of the black-box policy. Coppens et al. (2019) distill the black-box policy into a soft decision tree, a type of decision tree where the leaves output a static distribution over the actions and the inner nodes select the sub-branch using logistic model. A different approach is taken by Liu et al. (0019) which distill the model into linear model U-Trees, type of decision tree in which leaf nodes use linear model to produce their output (Q- values) instead of outputting a constant value. Both types of decision trees are more interpretable since they follow clear and simpler rules go down the tree and to pick the output value.
2

---

Under review as a conference paper at ICLR 2021
2.2 HIERARCHICAL REINFORCEMENT LEARNING
In Hierarchical Reinforcement Learning (HRL), an agent is composed of hierarchy of policies. The top layer decomposes the task into sub-tasks, the layer below decomposes sub-tasks into sub-sub- tasks, and on until the lowest level receives low-level task and attempts to solve it by interacting with the environment. Policies at higher layers learn to act at higher temporal and abstraction levels.
A subtask of can be defined in multiple ways, for example as simpler linearly solvable Markov Decision Problems (Earle et al. 2018) or as a tuple (Pi, Comp: R2) where the subtask of is eligible to start any time the precondition P. is satisfied and it is completed once the current state is part of the completion set Coomp upon which it receives reward r' ~ R, (Sohn et al. 2020).
Our approach is based upon goal-oriented hierarchical reinforcement learning, where completing task means reaching goal where that goal typically is state s which the agent must reach. Policies that receive a goal have only H steps to reach it instead of an unlimited time budget. The policy at the bottom of the hierarchy interacts with the environment while the other policies act by picking goals (i.e. their actions are goals for the policy below them). In some problem settings, the reward must be maximized. However, in other settings, the agent receives goal gene from the environment which must be reached as fast as possible. In that setting, it is important to note that the agent ignores the rewards produced by the environment; it only uses its internal reward scheme which gives the agent a small negative reward at each step, encouraging it to find short paths.
While goal-oriented hierarchical reinforcement learning has long history (Dayan & Hinton| 1992). there has been a resurgence in interest in recent years. Hierarchical-DO% (Kulkarni et al. 2016) combines hierarchical learning with deep learning for the first time; Hierarchical Actor-Critic (Levy) et al. 2017) improves performance by carefully setting up the hierarchy of actor-critic policies; Deep Feudal Reinforcement Learning (Vezhnevets et al., 2017) use abstract goals in a latent space instead of an actual state in the real state space S. More recently, Hierarchical Learning with Off-Policy Correction (Nachum et al.) 2018) tries to support off off-policy learning even though that all layers are constantly evolving by correcting the goals present in the transitions using heuristic method.
While goals produced by feudal networks (Vezhnevets et al., 2017) might be more effective for training. they do not fit our interpretability objectives either since the goal space is a latent space that is not directly understandable to researchers and non-experts.
3 GENERALIZED HINDSIGHT ACTOR-CRITIC WITH TEACHER
Our work builds upon the Hindsight Actor-Critic (Levy al., 2019) or HAC, state-of-the-an algorithm to train hierarchical agents, which achieves excellent performance in some environments beating the hierarchical agent algorithms mentioned before. We refer the interested reader to the original description of HAC (Levy et al. 2019) due to its non-trivial nature.
HAC is designed for a specific setting: enviromments that provide an end goal and where the only objective is to reach the goal as fast as possible. This specialization leads to 2 limitations: (1) HAC requires goal, making it incompatible with all environments which do not provide goal for the agent and (2) HAC ignores the rewards given by the environment since it uses an internal reward scheme. This makes it inapplicable to most environments, in which rewards can be given anytime.
To address these issues we generalize HAC, creating the HAC-Generaa with Teacher algorithm which doesn't require a goal and that considers the reward given by the environment. To avoid requiring a goal, the policy at the top of the hierarchy produces its output (a shorter-term goal) using only the state as input (no end-goal in the input). To take into account the rewards, the objective of the goal-picking policy becomes picking goals such that the maximum amount of reward is collected during the episode. The objective of policy at the bottom of the hierarchy (the goal goal-reaching policy) stays the same: reaching the short-term goal in most H steps, ignoring environment rewards.
3.1 MAINTAINING THE OPTIMALITY ILLUSION TO ADDRESS NON-STATIONARITY
These changes address the limitations of HAC, but they break HAC's technique to make training effective: addressing the non-stationarity problem, i.e. the problem that since all policies in the
3

---

Under review as a conference paper at ICLR 2021
Hierarchiical Agent
Pollcy
Goal
Level Policy
Environment
New state Reward
Environment
Figure I: Interaction between HAC-General hierarchical agent with 2 levels of policies and the environment. The goal-picking policy the top produces goals and the goal-reaching policy at the bottom interacts with the environment to reach those goals in most H steps. The environment produces reward and the state changes at each interaction. The objective of the goal-picking policy at the top is to pick goals that maximize the reward that is collected, while the objective of the goal- reaching policy at the bottom to reach the goal states and collects at least the desired amount of reward dictated in the goal.
hierarchy train in parallel, each policy needs to continuously adapt to the changes in the policies below it in the hierarchy (whose behavior relies on), which makes training difficult and unstable.
The insight of HAC is that if each policy trained under the illusion that all the policies below it were stationary, then would train faster and more efficiently. Since optimal policies are stationary, HAC attempts to give each policy the illusion that the policy below it is optimal and thus stationary. HAC carefully constructs types of transitions to create this illusion, where a transition a tuple of the form (state, action, reward, next state, goal, discount). While the 3 types of transitions are detailed in Appendix A for space reasons, we summarize how HAC creates the illusion that the policy below is optimal and how HAC-General With Teacher preserves that illusion.
We define some terminology: let π be the policy in the hierarchy for which we create the illusion. We call the picl-ni policy since it produces goals and call the policy below it in the hierarchy the goal-reaching policy Thelow- since it attempts to reach the goals it receives from π.
HAC. In HAC it is simple to give the illusion that Thelow is optimal because rewards only given when the goal-state reached. As shown in Figure [ if the action of π to pick the goal and the goal-reaching policy Thelow fails to reach and reaches state instead, w's action is replaced by the hindsight action s. The policy Thelow now appears optimal since picked goal s and Tbelow reached it.
Protend the policy picked pool
O
Bul alovet wrw.giitin
Reached
entien gualppicking golicy
concen coneeeeii Behavtor goal-raaching
Current
Behuvier palicy
(a) Original action
(b) Hindsight action
Figure 2: HAC's optimality illusion: the goal-reaching policy Thelow doesn't reach the original action/goal, g but after replacing the original goal/action by the hindsight action it appears optimal.
Problem. This technique breaks down when environment rewards matter and must be maximized, L.e. it breaks down for HAC-General with Teacher. Replacing g by not enough anymore to give
4

---

Under review as a conference paper at ICLR 2021
the illusion that the goal-reaching Teelow acted optimally: while Xbelow reached state it might not have collected the maximum amount of reward possible. other words, there might be an alternative path to the same final state where more reward would have been collected (Figure E Since in most environments, it is impractical or impossible to determine the optimal path was taken (or what the optimal path is), cannot guarantee that below appears optimal.
Haximum reward path (optimal)
Goa sttte Reashed
Lower reward poths (su cobbooptima))
Figure Now that rewards matter, reaching the goal isn't enough guarantee optimality: the highest-reward path to the goal must be picked.
Solution. To address this issue, HAC-General uses new definition of goals. The new goals have components: a state & which must be reached and a minimum amount of reward Twin which must be collected. As shown in Figure 4 if the original action/goal (s, Tmmnn but the goal-reaching policy reaches instead s' and collects r' reward, then the goal-picking policy's action is replaced by the hindsight action (s', Thindsight)) where lhhindsight S creating again the optimality illusion. is important to note HAC-Genera creates the same types of transitions as HAC; the major change the way goals are defined.
Goal statta
Destred reward R: 200
Reached state S
Desied reward R:: 150
Goal State
Curret State
Coflected R":
Gument
(bward 150
Statd
(a) Original action
(b) Hindsight action
Figure 4: HAC-General re-creates optimality by re-defining goals as a (state, desired reward) pair. If replace the original action (G, R) the hindsight action (S,R) R'), the goal-reaching policy looks optimal again. Optimality is reached because optimality relaxed: the policy must collect at least the desired reward to appear optimal, but not necessarily the maximum reward possible.
The goal-pieking policy now has mechanisms to maximize the reward collects (1) pick goal states that lead to high rewards and (2) force the goal-reaching policy to take high- reward path to s by making the minimum reward threshold Twin as high as possible. The second point makes it possible achieve high reward in RL environments where the reward is also tied to the action, not just the state, since the goal-reaching policy will learn to pick actions that lead to the stal-state but also lead to high reward.
Advantages.
3.2 LEVERAGING THE BLACK- BLACK-BOX AGENT
We can leverage the black-bor box agent to train the explainable agent more quickly and efficiently by letting the black box agent act as a teacher which provides partial demonstrations to the explainable hierarchical agent (the student). We stress that our method differs from classical imitation learning in two ways. First, the classical imitation learning algorithms are designed for non-hierarchical agents and cannot be directly applied for training hierarchical agents. Second, while a teacher is used improve training, the objective that is being optimized our work is not mimicking the expert, because that would create only a superficial link between the high level policy and the low
Note: during the training process, the agent learn pick both good states and good desired rewards

---

Under review as a conference paper at ICLR 2021
level policy. Instead, all policies are still only optimized to maximize the reward (and the same reward scheme as before) but we occasionally create transitions from the expert's behavior.
Since the goal of this paper is to create explainable agent, there is requirement that it should be trained from scratch with no help, hence leveraging black-box agent does not contradict our chosen setting.
During an episode, the poal-picking policy produces a series goals and the goal-reaching policy tries to reach each of those goals. To train the producing policy of the hierarchical agent, need to obtain goals from the black box expert but the expert doesn't produce any goals explicitly since it is not hierarchical agent. However, produces goals implicitly: if we let the expert act for roughly H steps, the last states it reaches are good goals. Additionally, the expert reached those final states using series of actions a1,..., an that can be used to train the goal-reaching policy.
Therefore, during an episode, each time a new goal needs to be picked, we stochastically decide whether the hierarchical agent or the black-box expert acts If the black-box expert chosen, acts for a number n [0.75H, H] of steps, reaching the state and collecting a total reward rn- Given the last state we can create goal Ghindsight = (Sn: Thindsight) where l'vnndightt S Tn- The algorithm then proceeds as usual, creating the transitions if it was the goal-picking policy in the hierarchy that had picked the goal. To train the goal-reaching policy, for each action a taken by the expert agent while it was reaching for sn create transition if had been the goal-reaching policy acting with the short-term goal ghindsight.
By interleaving the use of the hierarchical agent and the black-box expert during an episode, the behavior cloning problem is avoided [Ross Bagnell 2010)p which ensures the hierarchical agent achieve good performance even in states outside the expert innuuedd state distribution, i.e. it is in state the expert would rarely or never visit when it interacts with the environment by itself (without the intervention of the hierarchical agent).
Algorithm D shows the pseudo-code to train a specific level (Appendix B contains the full pseudo- code). Note: the 3 types of transitions created in the algorithm are detailed in Appendix A
Algorithm 1: Level function of HAC-General (With Teacher)
Input: Initialized hierarchy of actor-critics with & levels
Input: Teacher policy x* and probability C of using Function TrainLevel (level I, state s, goga g, x C):
St. It. cumulR + s.g.
repeat
Decide will test action/subgoal and then pick action a; if >> 0 then
then
9
8; r. Low.loowe//TTansiioonss +- expertRollout(ss.
+ [s reduceReward
11
Replay Buffer, < lowLeve/
12
else
13
8/ r + TrainLevel (1 - 1, 34 a4 T²,C))
14
else
15
Exccute primitive action ag, observe next state so and reward r
17
cumulR R < cummuR +
Create subgoal testing transition if failed to reach the subgoal and testing subgoals
19
Create the hindsight action and the hindsight action transition
20
Create incomplete hindsight goals transitions and put them in the HER buffer
21
8; +
22
until episode ended OR (H steps done or until any goal from above gn.tgeck reached);
24
Complete hindsight goal transition in buffer using HER and move them to the replay buffer Return the final state s; and total reward cumauIR

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
The goal of our experiments is evaluate the effectiveness of the HAC-General With Teacher algorithm to train goal-producing hierarchical agents. We do not compare against most classical reinforcement learning algorithms and against imitation learning algorithms. since the former cannot train hierarchical agents and the latter creates no guarantee that the goal-reaching policy actually attempts reach the goal, therefore not adding the interpretability we seek.
We compare 3 different algorithms: HAC-General with Teacher, HAC-General without Teacher, and the original HAC algorithm (to which we manually provide the end goal). We then use the trained hierarchical agent to generate explanations, which are visualized, showing the agent's 's plan to collect as much reward as possible.
We evaluate the agent in 2 enviromments from the OpenAl Gym (Brockman et al. 2016) which present different challenges: Mountain Car Continuous and Lunar Lander Continuous. In Mountain Car, car has to reach the top of a mountain to its right but does not have enough power to reach it directly; it has to move left and then leverage momentum to climb the right mountain. The challenge of Mountain Car is good exploration since it only gets positive reward at the top of the mountain but random exploration will very rarely lead to it; for other actions, it is punished through small negative rewards. In the Lunar Lander environment, lander has stabilize and land on the ground softly. This environment has more complex dynamics than Mountain Car, higher dimensional states (which makes picking good goals harder), and more complex actions for controlling the engines. The main challenge is predicting good states and desired rewards for the short-term goals, and learning how to reach those high dimensional goal states.
Figures 5 compares the performance of HAC, HAC-General Without Teacher, and HAC-General With Teacher. HAC achieves good performance in the Mountain Car (likely by knowing the end goal in an environment where exploration is the challenge) but fails to solve the more complex Lunar Lander environment (despite knowing the end goal). HAC-General Without Teacher only solves the Mountain Car environments on 3 out of 5 runs but always fails Lunar Lander. HAC- General With Teacher can solve both environments, beating the other hierarchical algorithms and showing how effective leveraging black-box expert can be to train the explainable hierarchical agent.
Mountain Car Evolution of Reward
Lunar Lander
Evolution Reward
100
250
W
Episode Reward
Episode Reward
-250
-500
-750
-50
HAC-Ganwal With
-1000
HAC-General With Teacher
HAC General Without Teadher
HAC General Without Teachor
-75
-1250
Envtronment success threshole
Environ rommont su succoss threshaid
250
500
750
1000
1250
1900
1750
2000
2000
4000
800C
10000
Episode
Episode
(a) Mountain Car
(b) Lunar Lander
Figure 5: Evolution of the performance of the agent during training in the Mountain Car and Lunar Lander environments. Each algorithm is executed 5 times per environment: the bold line represents the mean reward and the shaded regions display the standard deviation. HAC-General With Teacher performance exceeds the success threshold defined by the environment and outperforms both com- peting algorithms, showing both its effectiveness despite being in a more general and harder setting than HAC and the usefulness of leveraging black box expert during training. Note: training stops once the success threshold is reached, since it indicates the agent can solve the task.
7

---

Under review as a conference paper at ICLR 2021
4.1 CREATING GOAL-BASED EXPLANATIONS
Given trained hierarchical agent, we can leverage its goal-picking component to explain the agent's actions and future behavior. The current goal explains the short-behavior of the agent since it reveals the state the agent is trying reach. We can obtain more useful explanations by querying for series of goals instead only knowing the current goal. These goals form the agent's s plan to solve the environment and collect the maximum amount of reward; the goals help the understand the agent's 's present and future behavior. To obtain this series of goals, the goal-picking policy is queried repeatedly: given the agent's state 5, we query for the current goal 9l ( ri); then assume the agent reaches the state 81 and query for the next goal Ø2 = (82, r2); this process repeats until the desired amount of goals has been collected. Figure [ displays the goal-based explanations at different time-steps of an episode, for both environments.
(a) Mountain Car
(b) Lunar Lander
Figure 6: Explanations produced by the hierarchical agent different time steps of episode. The red or yellow rectangle indicates the current goal. A plan composed of goals 92, is generated given the current state and shown as progressively darker green rectangles. video version of the explanations is provided in the suppicmentary material.
In experiments, the predicted goals reached the majority of the time by the goal-reaching policy (see Appendix [C), thus making the plans reliable indicators of the future behavior of the agent and useful interpretability tools.
5 PROPERTIES OF THE INTERPRETABILITY METHOD
Dealing with failures of the goal-reaching policy. One advantageous property the agent is that the plans can be created from any state s. Therefore, if at any point the goal-reaching policy fails to reach the goal by large amount, interpretability will suffer since all the goals the plan that come afterwards may not be valid anymore. However, performance may still be good because the goal-pree produccnd policy receives the current state when picking a goal and so can adapt to the errors made by the low-level policy.
Why not simply run the simulator? While simply running the simulator also indicates the behav- ior of the agent, our method has advantages. It is for example advantageous in stochastic settings: we know at any time what the agent expects to be the best set of goals (before the stochastic event has happened) and understand how it adapts its the future goals as stochastic event happen and the agent ends up in states outside its initial goal plan.
Orthogonality to other interpretability methods. The hierarchical agent add some interpretabil- ity due to the goals it produces, but a natural question arises: aren't the policies hard to interpret too? The strength of our method is that it is orthogonal to other interpretability techniques and can be combined with them and future progress the field. Any method that can add interpretabilityy to the neural networks underlying the policies could be applied the policies of the hierarchical agent, making it possible not only to interpret better how the agent picks environment actions, but also to interpret better how high-level goals are picked, which might bring additional insight then in
8

---

Under review as a conference paper at ICLR 2021
hierarchical agent. However, our method already brings some interpretability even without applying those other interpretability techniques; they complement each other instead of being in opposition.
Goal granularity. The number of intermediary goals is controlled through the maximum number of actions H the goal-reaching policy may do to reach the goal before a one is picked by the higher level policy picks a new one. A low value for H means goals are fine-grained and should be rapidly reached, while high values for H may lead to goals which require many more actions to reach and are therefore less granular.
6 CONCLUSION
In this paper, we tackle the problem of understanding the decisions taken by deep RL agents, which tend to be inscrutable black-boxes. We develop a new technique to understand agents, goal-based interpretability, in which the agent expliciily produces a series of goals and attempts to reach those goals one by one, collecting as much reward as possible along the way. These goals make the agent more explainable since the agent's current behavior is clear (reach the next goal) and the user knows the agent's s long-term plan for solving the task. The agent becomes more interpretable since we know any time the goals it seeks to reach and how this plan adapts as stochastic events happen in the environment. Additionally, obtaining the plan does not require taking any actions, contrarily to simulations, which is useful real-world applications.
To create the goals, we rely on 2-layer hierarchical agent, where the top layer produces goals and the bottom layer attempts to reach these goals. We contribute a new algorithm designed to train hierarchical gool-prrouuing agents, which we call HAC-General With Teacher, which generalizes the Hindsight Actor-Critic (HAC) algorithm. Contrarily to HAC, HAC-General attempts to maxi- mize the reward collected from the environment and doesn't 't require the enviromment provide an end goal the agent must reach. HAC-General is also able to leverage black-box agents to improve the training process of the hierarchical agent. We detail possible improvements in Appendix D and note that the algorithm can be adapted to accommodate different definitions of goals, allowing for further research.
Our experiments show that HAC-General improves on HAC and can train goal- proouucoo hierarchi- cal agents faster and where HAC may fail train. The trained hierarchical agent produces reliable and clear explanations which can be visualized, helping users understand the agent's 's short-term and long-term behavior. Additionally, our technique can be combined with other interpretabilty methods, adding more insight into how goals and environment actions are chosen.
REFERENCES
Adrià Puigdomênech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benehmark. CoRR, hass/2003.13350, 2020. URL https: http::///axvvvvoo org/ httpg//bb//200330 2003.1335C
John Bragg and Ibrahim Habli. What acceptably safe for reinforcement learning? In Barbara Gallina, Amund Skavhaug, Erwin Schoitsch, and Friedemann Bitsch (eds.), Computer Safety, Reliability, and Security SAFECOMP 2018 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Västerås, Sweden, September 18, 2018, Proceedings, volume 1094 of Lecture Notes in Computer Science, pp. 418 430. Springer, 2018. doi: 10. 0000077788-3119992229 .5.5. URL
https:://oi..om org. 10. 1007/978 319 99229-9 1-229-7-33 7_35
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01.5 2016.
Youri Coppens. Kyriakos Efthymiadis. Tom Lenaerts, and Ann Nowe. Distilling deep reinforce- ment learning policies soft decision trees. Proceedings of the IJCAI 2019 Workshop on Expluinable Artificial Intelligence, pp. 1-6, 8 2019.
Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In Stephen Jose Hanson, Jack D. Cowan, and C. Lee Giles (eds.), Advances in Neural Information Processing Systems 5,

---

Under review as a conference paper at ICLR 2021
[NIPS Conference, Denver, Colorado, USA, November 30 December 3. 1992], pp- 271-278. Morgan Kaufmann, 1992.
Adam Christopher Earle, Andrew M. Saxe, and Benjamin Rosman. Hierarchical subtask discovery with non-negative matrix factorization. In 6th International Conference on Leurning Representa- lions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceed- ings. OpenReview.net, 2018. BRLLhtps::/ http:///ppeeeeiiieeee /ptptttps//peeeeeeeeee
net worummruiii coorumiidrrp WMWMOW
Maria Fox, Derek Long, and Daniele Magazzeni. Explainable planning. CoRR, abs/ 1709.0026, 2017.
Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res., 16:1437-1480, 2015. URL (nttp://dl. aw...accm.oo wwt.:o/gg//./ioooeo cfm?
id (d-2886795
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssunn Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1787-1796. PMLR, 2018.
Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia P. Sycara. Trans- parency and explanation in deep reinforcement learning neural networks. In Jason Furman, Gary E. Marchant, Huw Price, and Francesca Rossi (eds.), Proceedings of the 2018 AAAI/ACM Conference AI, Ethics, and Society, AIES 2018, New Orleans, LA, USA, February 02-03, 2018, pp. 144-150. ACM, 2018
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Sacedi, and Josh Tenenbaum. Hicrarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gamett (eds.), Ad- vances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3675-3683, 2016
Andrew Levy, Robert Platt Jr., and Kate Saenko. Hierarchical actor-critic. CoRR, abs/ 1712.00948 2017.
Andrew Levy, George Dimitri Konidaris, Robert Platt Jr., and Kate Saenko. Learning multi-level hierarchies with hindsight. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, 2019.
Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingean Li. Toward interpretable deep reinforce- ment learning with linear model u-trees. In Michele Berlingerio, Francesco Bonchi, Thomas Gärtner, Neil Hurley, and Georgiana Ifrim (eds.), Machine Learning and Knowledge Discovery in Dutabases, pp. 414-429, Cham, 2019, Springer International Publishing. ISBN 978-3-030- 10928-8.
Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. bede learn- ing: A survey. CoRR, abs/2006. 16712, 2020. URL httpp://aaxxlv.oor//a///00000
16712
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, Montréal, Canada, pp. 3307-3317, 2018.
Erika Puiutta and Eric M. S. P. Veith. Explainable reinforcement learning: A survey, CoRR, abs/2005.061 066277, 2020.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Yee whye Teh and D. Mike Titterington (eds.), Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pp. JLMR, 2010.
10

---

Under review as a conference paper at ICLR 2021
Julian Schrittweser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. CoRR, abs/191 1. 108265, 2019. URL
08265
Sungryull Sohn, Hyunjae Woo, Jongwook Choi, and Honglak Lee. Meta reinforcement learning with autonomous inference of subtask dependencies. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
URL / net forumm did
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. CoRR, abs/1703.01161, 2017.

---

Under review as a conference paper at ICLR 2021
A TRANSITIONS CREATED IN THE HINDSIGHT-ACTOR-CRITE ALGORITHM
Figure summarizes the transitions and we detail them below:
1. Hindsight action transitions: consider the scenario where the policy я at level picked as action the goal ay = and the policy below =t-1 reached state s'. If s' is close enough to s. then the goal was reached and the policy below Th-1 acted optimally. If the goal wasn't 't reached, the action taken by the policy xı at level I is substituted by the hindsight action Ahindsight = s'. Under this scenario, the policy below now acted optimally, since it reached the updated goal.
2. Hindsight goal transitions: for a policy ny at level I, while hindsight action transitions give the illusion that the policy below Wh-1 is optimal, the reward signal may still be extremely sparse if the policy Ni never reaches the goal it received from above as input. To fix this issue, new transitions are created by replacing the goal gi of the hindsight action transitions by one of the states reached by policy while it was doing actions. This ensures that during least one step of the episode the policy gets a positive reward since the hindsight goal is one of the states it visited.
3. Subgoal testing transitions: in the two previous types of transitions, the hindsight action is always state that the policy below reached. Therefore, there are no transitions in which the action is a goal the policy below didn't 't reach in at most H steps, which makes it impossible for the policy Ti to learn which states are unreachable in H steps. These unreachable states should not be picked, because then the policy below TT-1 can never reach the goal it received, which will considerably slow down its training process and may lead to erratic and unstable behavior. To avoid picking unreachable goals actions, we want to assign very low Q-values to those actions. However, there is one key uncertainty: if the policy Ii produced the goal gi but the policy below Ti-1 reached the state instead, it might be either because the policy z hasn't 't learned to reach the goal or because Ii is unreachable in most H steps. Since it subgoal testing transitions are created a random fraction of the time (say 20%). In these transitions, the raw action is used (instead of the hindsight action) and very low reward is attributed to that action, to avoid picking that goal again in the future.
possible to distinguish between the cases with certainty,
Note: for the bottom-level policy, we only create hindsight goal transitions (using the original ac- tion). Since the bottom-level policy interacts with the environment instead of producing goals as actions, it is not possible to create hindsight action transitions and subgoal testing transitions.
Hindsight action
Hindsight goal transition
Subgoal testing
transition
transition
original goal
hindsight goal
original goal
Policy level
Policy at level L
Policy at level L
hindsight action
hindsight action
original action
Policy at leval L.
Policy at level L. 1
Policy at level L-1
Figure 7: The three types of transitions created by the Hindsight Cctor-Critic algorithm: hindsight action transitions (replace the action by the state that was reached by the policy below), hindsight goal transition (replace action; replace the goal by a state reached by the policy during the episode) and subgoal testing transition (if the policy at level L picks action/goal the policy below fails to reach, occasionally attribute that action a very low reward).
12

---

Under review as a conference paper at ICLR 2021
B PSEUDOCODE FOR THE
WITH TEACHER ALGORITHM
The pseudo-code of the training loop and a more detailed version of the TrainLevel function is shown below. The HAC-General without Teacher algorithm can be obtained by setting C'
The full implementation can be consulted in the supplementary material. The supplementary ma- terial also contains the implementation the teacher agents as well as the code to show visualize the goals in the environment. For both environments, GIF file shows the trained hierarchical agent acting, and both the current goal and future goals are visualized at all steps.
Algorithm 2: Train- (2-level hierarchy)
Input: Level
Input: Start state and goal
Input: Expert policy π²
Input: Probability C of using the teacher
Output: Final state 8; total reward cumul
Si, Dé, R +- 8, 9,
repeat
10
Decide will test action/subgoal
a2 (t) (no noise noise if subgoal testing)
12
if then if unseTeccchrinntteratiiCCCCCCCC then
14
8, r, www.wevveTraansiiooss + expertRollout(s, n*)
a; [s reduce
17
Replay Buffer, <-
18
else
r + TrainLevel (i 1, Qj, ",, C)
29
else
Execute primitive action dp Observe next state 8g and reward R cumuu +r
22
23
24
25
if > and subgoal a; was tested and wasn't reached then
26
Replay Buffer, + (ss, 04, Penalty, $;. = 0] // Subgoal testing transition
28
> and a; wasn't reached then [ reduceReward((r) // Hindsight action
29
Rs geeTransttoonRewardd cummu cumul oz)
31
Replay Buffer, f- [ss, 0,, R₁, s, 96. 7 {r, 0}] // Hindsight action transition
33
// Hindsight goal transition (step 1) it' not isTopLeve((i) then
35
36
HER; + [St, 0,, TBD, s. TBD,, TBD)
37
38
81 1
39
until episode ended OR (H steps done or until any goal from above 9n.ign<k is reached);
41 Hindsight goal transition (step 2): perform HER; complete r, and r
42 Replay Buffer, Perform HER(HER,)
13

---

Under review as a conference paper at ICLR 2021
Algorithm 3: HAC-General (With Teacher)
Input: Initialized hicrarchy of actor-critics with levels
Input: Expert policy x²
Input:
Probability C using the teacher
for M episodes do )
TrainLevel (k 1, goal N None, TT,
Update all the actor-critic networks in the hierarchy
end
C PERCENTAGE OF GOALS THAT ARE REACHED
Relative frequincyy
Average Median
0.6
Relatve frequnny
0.6
Fraction 0.6
(a) Mountain
Relative fruquinccy
Average Median
0.8
Relalve freqanny
0.6
Fradtion gosls
spisode
(b) Lunar Lander
Figure 8: The trained hierarchical agent for hundred episodes and for each episode mea- sure the percentage of goals that were reached during the episode. The distribution of those percent- ages makes it possible to better understand how often the goals reached. In the Mountain Car environment the goal is almost always reached (92 % median, 92% average) and in the Lunar Lander environment the goal reached the majority of the time (96 % median, 74% average) despite goals being much more difficult to pick and reach.
14

---

Under review as a conference paper at ICLR 2021
D POSSIBLE IMPROVEMENTS
While we displayed explanations generated by the good-performance hierarchical agents and saw their effectiveness gaining insight into the agent's makiing mechanism and future behav- ior, improvements could be made:
1. While we generated plans at different time steps to validate them, this does not prove the plans are always effective (this is similar to the principle that tests can show the presence of bugs but cannot prove an absence of bugs). Finding mechanisms to further verify the reliability of the explanations would be interesting future work.
2. Work could be done verify that the assumptions behind the plan- -anneerrrt routine are met. Experiments show that the goal is reached the vast majority of the time for both the Lunar Lander and the Mountain Car environment (Figure (). but more detailed examina- tion could be useful. Additionally, we could also verify that the goal-picking policy isn't t sensitive to small changes to the state it receives as input.
3. Training hierarchical agents in the setting where no goal is provided and the agent attempts to maximize the collected reward is a challenging task. Developing better algorithms to train hierarchical agents would make it possible to tackle more difficult environments and improve the efficiency and speed of the training process.
4. The experiments done in this paper did not deal with very high-dimensional states such images. Predicting good states is likely very challenging in such environments and determining whether a goal has been reached or not becomes more difficult question, so new techniques need be developed for those scenarios, such as re-defining goals so that only the relevant part of the state is included in the goal (and background visuals are ignored, for example).
E HEURISTIC TO DETECT THE END OF A PLAN
One heuristic determine how many goal-states should be queried when creating the plan to detect when the goal-states stabilize. For example, when the lunar lander has reached the ground, the next goal should to stay in the same state. Similarly, the mountain car reaches the top the mountain, the next goal should be to stay at the top of the mountain. Therefore, if we detect that the state-goal& s' produced by the goal-picking policy is very similar to the current state s, we may stop querying for new goals.
15


Reviews:
[{"review_id": "ta_Zlj_R2cS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a hierarchical RL method where the high level controller produces a series of sub-goals in an open-loop fashion which the low-level controller attempts to reach sequentially with the aim of maximising task rewards. The agent is trained using an extension of Hindsight Actor-Critic (HAC) algorithm. The algorithm also leverages a model-free flat policy trained on task rewards as an expert. The approach is evaluated on two tasks: Mountain Car and Lunar Lander. \n\nThe related work section lacks a thorough analysis of existing literature in this domain. Additionally, the lack of baselines is quite concerning, I would have liked to see a comparison with closed loop hierarchical agents, or model based hierarchical agents.\n\n1. While the motivation of the work is interesting, the paper lacks novelty and fails to provide any comparisons with existing HRL methods that could also directly fit in the category of explainable RL methods. There have been several papers attempting sub-goal discovery in HRL which are not discussed in this work. Hence, it is hard to assess the novelty and how it stands in comparison to existing methods. Some recent work in this direction: ([Jurgenson et. al., 2019](https://arxiv.org/abs/1906.05329), [Parascandolo et al., 2019](https://arxiv.org/abs/2004.11410), [Nasiriany et al., 2019](https://arxiv.org/abs/1911.08453))\n\n2. To my understanding, the method only works if the low-level controller has near optimal performance as otherwise setting open-loop goals would not be plausible. For example, the agent might diverge significantly from the path expected by the high level-controller while reaching one of the subgoals and hence the rest of the plan is not relevant anymore. This limits the usefulness of this method significantly, making it only plausible in deterministic settings and it is unlikely that it would generalise outside of the distribution of data seen during training. Could you comment on how this could be addressed?\n\n3. Connected to that, other work on model-based HRL also seems directly relevant in this context where there is no need to assume necessarily that the low-level controller is perfect as the high level transition model is learned and leveraged during planning to find a most rewarding series of subgoals. This has an additional benefit of not assuming the low-level policy is perfect and rely on a learned transition model of behaviour for the agent. It also does not rely on having access to an expert low-level controller to provide demonstrations. It would be nice to better motivate the reasoning behind assuming a perfect low-level controller and discuss its failure cases. Relevant works in this direction include  an ([Nasiriany et al., 2019](https://arxiv.org/abs/1911.08453 ), [Pierrot et al, 2020](https://arxiv.org/abs/2007.13363))\n\n4. It is not clear to me how many goals are queried from the high level controller at the start of an episode, the paper states “...,this process repeats until the desired amount of goals has been collected.” It would be good to discuss this in the main text.\n\nIn summary, while the motivation seems sound and important, the problem the paper is addressing is not particularly novel. At the same time, the assumptions made seem quite restrictive to me, the experiments are not particularly convincing and rigorous comparisons to baselines are missing.\n\nHence unfortunately, I don’t believe this paper, in its current form, is appropriate for publication. But I encourage the authors to improve their related work section, discuss and compare their method with existing approaches to subgoals discovery and model-based HRL to better motivate the novelty and significance of this work.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ktKWc9Ei5LZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a hybrid imitation learning/reinforcement learning method for learning hierarchical policies where the top layer provides sub-goals and desired cumulative rewards and the bottom layer learns to meet these goals. The advantage of such a decomposition is interpretability of the learned policy. The algorithm is evaluated on MountainCar and LunarLander from OpenAI’s gym. The authors show that their imitation learning/RL scheme is able to solve both tasks while producing reasonable sub-goals. \n\nI found the sub-goal for interpretability idea very interesting. However the paper lacks in clarity when presenting the algorithm and in depth when evaluating it. For the presentation, the algorithm box should be commented and its main components should be explained in the main paper. Since most of the learning is carried away by HER (which is not even properly referenced in the algorithm box), a background section should be added. The paper should aim to be more self-contained. The idea of adding the cumulative rewards as a goal is easy to grasp and its explanation can be shortened, leaving space for a more thorough explanation of the actual learning of the hierarchical policy. \n\nFor the experiments part, important details are left out. For instance what is the ratio of imitation learning and reinforcement learning? How much imitation learning is needed for learning the task? How is the black-box policy obtained? What is its performance? To which extent the low level policy actually depends on the goals given by the higher level policy? Since imitation learning is involved for both layers, one could imagine that the top layer is learning to produce reasonable goals, the lower layer is learning to produce reasonable policies but there is only a weak connection between the two, and the lower level is only giving the illusion of tracking the top level’s goals.\n\nI’m also unsure about some of the motivations. It is stated in the introduction that the drawback of black-box policies is that they can have surprising behaviors when facing unexpected states. How is the proposed policy immune to this phenomenon? \n\nFinally, the number of intermediary goals should be discussed somewhere. In robotics, it is common to define the policy as generating a trajectory in joint space and to let a lower level position controller track the trajectory. One can also easily modify OpenAI’s Mujoco tasks to be position controlled and not torque controlled. Would a policy defined in this MDP, where an action is a desired joint configuration of the next state, be de facto interpretable, or is there some constraint in keeping the number of subgoals small to be human readable? \n\nOverall, I like the direction and I think the plots showing the produced subgoals are very encouraging. However many details of the algorithm are left out and the experiments are not thorough enough to be fully convincing.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fJ-UxVQfKuS", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a hierarchical reinforcement learning algorithm in an attempt to improve the explainability of RL agents via the goals proposed by the higher level policy.  \n\nStrengths \n\nThe problem of designing safer and more transparent RL agents is an important and rather neglected one, so it is nice to see papers on this topic.  \n\nWeaknesses \n\nHowever, I think the paper makes some unsubstantiated claims, lacks thorough empirical evaluations, and is not very well-motivated and situated in the broader RL literature. \n\nFirst of all, the proposed approach is only evaluated on two very simple tasks (Mountain Car and Lunar Lander) and it is only compared with two ablations of the proposed method. At the very minimum, the paper should include comparisons with a strong RL baseline (i.e. SAC, PPO etc.), a strong HRL baseline (i.e. feudal networks), and a behavioral cloning baseline (i.e. vanilla BC, Dagger, or GAIL) since the algorithm includes elements of all these. I would be quite surprised if SAC and BC don’t achieve similar or much better performance than the proposed methods on these simple environments. Given that you have access to an expert, it is really unclear to me why you would want to use your method versus simple behavioral cloning. There needs to be better  motivation. \n\nI also do not think the choice of environments is well-motivated. Besides the fact that they are quite simple for current SOTA, they do not strike me as particularly good for emphasizing explainability or for using a hierarchical agent (since they can be easily solved without HRL). I also don’t understand how the goals are useful in this particular case. Given that these are fixed environments, once you’ve trained a RL policy you can accurately predict the agent’s trajectory including all the visited states rather than only some of them (as the proposed method does). I think a more interesting use case would be to test the method on a new environment and show the goals selected by the higher level policy since a typical RL method wouldn’t be able to predict what states it will visit in a new environment without a simulator or running the policy. I suggest using environments that are better suited for your goals and approach and where other methods might fail or would be hard to explain. \n\nIn section 4.1 you make claims regarding the “resistance to small errors” and to “dynamics” / stochasticity which are not supported by any experiments or theory. I suggest either removing them or backing them up with some empirical evidence. \n\nFinally, I think the core idea of the paper needs to be better motivated. After reading the paper, I am not fully convinced that the proposed architecture to generate goal-based explanations is very useful for understanding and supervising an agent’s behavior. If you consider the environment in which you are training, then I don’t see how predicting goals is better than simply using a standard RL to predict actions and use the simulator to predict future states. If you consider a new environment, I don’t see how the proposed method can generalize if the state space is different so it won’t know what goals to output or its predictions will be off. \n\nAlso, what is the level of granularity for generating the goals (i.e how often the higher level policy acts relative to the lower level policy) and is there a way to tune this? This seems like an important design choice which is not discussed very much in the paper. \n\nRecommendation\n\nGiven the above points, I do not think the paper is ready for publication. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cilAMfU5V-p", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript proposes (1) HAC-General with Teacher (GT), an extension of Hindsight Actor-Critic (HAC) that incorporates the environment reward as part of the goal formulation, and (2) \"goal-based explanations\", a framework in which the agent is tasked to produce intermediate goal states.\n\nGood things:\n\n+ The problem setting is important and interesting, and the paper lists a good number of applications where adding interpretability to policies would improve their applications to impactful real-life settings.\n+ The idea of extending HAC to goal-less environments is also interesting, as it could potentially shine a method for systematically utilise HRL algorithms on standard RL environments.\n\nConcerns:\n\n1. The paper overall doesn't flow well, which makes it difficult to evaluate the novelty and quality of the method.\n\n2. The concept of \"goal-based explanations\" is confusing. Most modern HRL work assumes the same problem decomposition, that that the agent needs to learn a goal-proposing policy (which might give additional signals such as rewards, embedding of the state, etc) and a acting policy that proposes environment-compatible actions. Thus it is unclear how this setup differs particularly from others.\n\n3. The paper doesn't do a good job at explaining how looking at a series of goals (which in this case are just states) may be interpreted, and how these would correspond to \"explanations\". Note that I do not wish to be nitpickinging about this particular naming choice, but \"explanations\" almost raises the expectation that the end user would see some sort of dense (and possibly language-based) description of the policy.\n\n4. GT builds on HAC, but HAC is never properly introduced. Section 3 can be mostly summarised as informing the reader that (a) HAC requires a hand-tuned goal proposal function, (b) that it doesn't utilise the environment reward (but it is not clarified why this would be a problem), and (c) that it has something to do with the problem of \"non-stationarity\". So, while section 3.1 does attempt to define then the differences between HAC and GT, it is extremely difficult to understand GT without clarifying how the manuscript intends HAC to look like.\n\n5. The terminology used in Section 3 is extremely confusing. What is a \"goal/action\" (or \"action/goal\")? What does it mean for a policy to reach it? What is the difference between a normal action and a \"hindsight action\"? It would be good if all of these things were properly defined, and not left only to graphical form in figures 2, 3, and 4 (which all look extremely similar, and not that clarifying).\n\n6. Section 3.2 declares that the goal of the work is to \"create an explainable agent\", and thus it is fair to use a pretrained teacher policy. However, (a) HAC does not require such a training setup, so it's unclear how to fairly assess whether GT is an improvement over it, and (b) it brings the method much closer to imitation learning, which the paper does not review at all. There are in fact methods that distill this sort of knowledge from teachers (e.g. https://arxiv.org/abs/1803.03835, https://arxiv.org/abs/1511.06295 https://arxiv.org/abs/2002.08037), and the proposed method looks extremely similar to them. It is also unclear how such a teacher policy would be trained to begin with, and how GT deals with e.g. probable sub-optimality of this teacher.\n\n7. The experimental setting can be greatly improved. The paper proposes to test GT against HAC on mountain car and lunar lander, but (a) both environments are extremely easy to solve, and (b the difference in performance is probably due to HAC being presented an extremely sparse reward setting.\n\nDue to these concerns, I currently cannot recommend acceptance, however I'd be willing to chance my score if the authors were to improve the manuscript by:\n\n- Providing a detailed explanation of HAC, and how GT fundamentally _improves_ it towards achieving better interpretability.\n- Providing a comparison (at least narrative-wise) against modern imitation learning literature.\n- Improve the experimental setup by:\n  a. At least having another modern RL algorithm as a baseline;\n  b. Providing a fairer adjustment to HAC (or at least making an argument on how it is currently fair);\n  c. Attempting a less trivial environment.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for hierarchical decision making where the intermediate representations between levels of the hierarchy are interpretable. I personally really like this general direction, as did most of the reviewers. Unfortunately, it was felt that, even after discussion, this paper is not ready for publication. To summarize the general spirit of the objection to this paper, all reviewers found that the experimental section was faulty and did not match the claims of the paper. Specifically, criticism here surrounded first, the choice of experimental setting, which was not considered to be the best for testing interpretable hierarchical decision making approaches; and second, the choice of comparison/baselines, which did not give sufficient security that the results produced by the proposed approach were sufficiently impressive.

I am satisfied that the reviewers considered the paper fairly, gave constructive criticism, and took onboard the author feedback. As a result, I am recommending rejection. I nonetheless think that the high quality feedback provided here will enable the authors to prepare follow-up experiments that may show their method in a more robust positive light, and encourage them to submit to a future conference, once armed with such results.
Decision: Reject