Explainable Reinforcement Learning Through Goal-Based Interpretability

1 INTRODUCTION . Deep learning has had a huge impact on Reinforcement Learning , making it possible to solve certain problems for the first time , vastly improving performance in many old problems and often exceeding human performance in difficult tasks ( Schrittwieser et al. , 2019 ; Badia et al. , 2020 ) . These improvements come at a price though : deep agents are black-boxes which are difficult to understand and their decisions are hard to explain due to the complexity and non-obvious behavior of neural networks . In safety-critical applications , it is often fundamental to check that certain properties are respected or to understand what the behavior of the agent will be ( Garcı́a & Fernández , 2015 ; Bragg & Habli , 2018 ) . Simply observing the behavior of the agent is often not enough , since it might take its actions for the wrong reasons or it might have surprising behavior when faced with an unexpected state . Ideally , the agent would explain its behavior , which would allow for auditing , accountability , and safety-checking ( Puiutta & Veith , 2020 ) , unlocking the use of Reinforcement Learning systems in critical areas such as robotics , semi-autonomous driving , or industrial applications . We provide three contributions to make more interpretable deep agents . First , we develop a new type of explanation for the agent ’ s behavior . Imagine the following scenario : a robotic agent has to traverse a difficult terrain until it reaches a specific building , where it collects a reward . The agent decomposes its task into a series of goals ( for example , positions it has to reach ) and tries to reach these goals successively until it reaches the reward zone . The agent is more interpretable since it explicitly produces the successive goals it is trying to accomplish : the current goal explains its shortterm behavior ( the joint movements are done to reach the current goal position ) and the remaining goals help us understand the agent ’ s overall plan to solve the task and predict its future behavior . We call goal-based explanation or goal-based interpretability the use of plan composed by a series of goals . Both model-based reinforcement learning ( Moerland et al. , 2020 ) and planning techniques ( Fox et al. , 2017 ) appear similar to goal-based explanations but there are important differences that make this technique novel . Goal-based explanations do not require learning a model of the environment ( neither the reward function nor the transition function ) , thus being compatible with both model-free and model-based reinforcement learning . Planning can be a useful explainability technique , but it has a few limitations : it typically requires knowing the end goals , they often can not be applied to complex Markov Decision Problems and they may have difficulty handling very large or continuous action spaces or state spaces . Our approach suffers from none of these limitations . Second , we develop a method to make the agent produce the goals that add interpretability . To do so , the agent is structured as a 2-level hierarchy of policies , with a goal-picking policy that produces goals and a goal-reaching policy that attempts to reach them . Goals are ( state , minimum desired reward ) pairs , meaning the goal-reaching policy has to reach a specific state in at most H steps and collect a minimum amount of reward along the way . To create a goal-based explanation , the goal-picking policy is queried repeatedly : given the agent ’ s state s , we query for the current goal g1 = ( s1 , r1 ) ; we then assume the agent reaches the state s1 and query for the next goal g2 = ( s2 , r2 ) ; this process for a fixed amount of steps per environment , though in future work more sophisticated algorithms to determine the adequate number of goals could be compared . Our third contribution is developing HAC-General , a new algorithm specifically designed to train goal-producing hierarchical agents . This algorithm builds upon the Hindsight Actor-Critic ( HAC ) algorithm ( Levy et al. , 2019 ) and makes it more widely applicable by not requiring the environment to provide an explicit end-goal . Instead of trying to reach the end-goal as fast as possible and ignoring the environment ’ s rewards , the HAC-General algorithm trains the agent to maximize the collected reward . Our extension tries to preserve the key property that makes the Hindsight ActorCritic algorithm effective : having an effective strategy to deal with non-stationarity by giving the illusion that the policies in sub-levels are optimal . The HAC-General algorithm is also able to leverage a black-box expert to improve and speed up the training for the hierarchical agent . 2 BACKGROUND & RELATED WORK . 2.1 EXPLAINABLE REINFORCEMENT LEARNING . The Reinforcement Learning community has recognized the need for interpretable and explainable agents , and researchers have developed several methods to add explainability and interpretability . Puiutta & Veith ( 2020 ) survey explainability techniques ; we briefly describe some key methods . To add interpretability , saliency-map methods determine the importance of each input feature for the policy when it generates its output . Perturbation-based methods ( Greydanus et al. , 2018 ) measure importance by perturbing different parts of the input and measuring the change in the policy ’ s output . The larger the change in output , the more important the feature ; the magnitude of the change quantifies the relative importance of features , making it possible to build the saliency map . In object-based saliency maps ( Iyer et al. , 2018 ) , in addition to measuring the importance of raw features , they also measure the importance of the whole objects present in the image . The importance of each object is measured by masking it and measuring the change in the policy ’ s output . Thus , a higher-level object saliency map is created which can be more easily interpreted by non-experts . Another approach is to distill the policy of the black-box agent into a simpler , more interpretable model while trying to preserve the behavior and performance of the black-box policy . Coppens et al . ( 2019 ) distill the black-box policy into a soft decision tree , a type of decision tree where the leaves output a static distribution over the actions and the inner nodes select the sub-branch using a logistic model . A different approach is taken by Liu et al . ( 2019 ) which distill the model into linear model U-Trees , a type of decision tree in which leaf nodes use a linear model to produce their output ( Qvalues ) instead of outputting a constant value . Both types of decision trees are more interpretable since they follow clear and simpler rules to go down the tree and to pick the output value . 2.2 HIERARCHICAL REINFORCEMENT LEARNING . In Hierarchical Reinforcement Learning ( HRL ) , an agent is composed of a hierarchy of policies . The top layer decomposes the task into sub-tasks , the layer below decomposes sub-tasks into sub-subtasks , and so on until the lowest level receives a low-level task and attempts to solve it by interacting with the environment . Policies at higher layers learn to act at higher temporal and abstraction levels . A subtask φi can be defined in multiple ways , for example as simpler linearly solvable Markov Decision Problems ( Earle et al. , 2018 ) or as a tuple ( P i , Cicomp , Ri ) where the subtask φ i is eligible to start any time the precondition Pi is satisfied and it is completed once the current state is part of the completion set Cicomp upon which it receives a reward r t ∼ Ri ( Sohn et al. , 2020 ) . Our approach is based upon goal-oriented hierarchical reinforcement learning , where completing a task means reaching a goal where that goal typically is a state s which the agent must reach . Policies that receive a goal have only H steps to reach it instead of an unlimited time budget . The policy at the bottom of the hierarchy interacts with the environment while the other policies act by picking goals ( i.e . their actions are goals for the policy below them ) . In some problem settings , the reward must be maximized . However , in other settings , the agent receives a goal genv from the environment which must be reached as fast as possible . In that setting , it is important to note that the agent ignores the rewards produced by the environment ; it only uses its internal reward scheme which gives the agent a small negative reward at each step , encouraging it to find short paths . While goal-oriented hierarchical reinforcement learning has a long history ( Dayan & Hinton , 1992 ) , there has been a resurgence in interest in recent years . Hierarchical-DQN ( Kulkarni et al. , 2016 ) combines hierarchical learning with deep learning for the first time ; Hierarchical Actor-Critic ( Levy et al. , 2017 ) improves performance by carefully setting up the hierarchy of actor-critic policies ; Deep Feudal Reinforcement Learning ( Vezhnevets et al. , 2017 ) use abstract goals in a latent space instead of an actual state in the real state space S. More recently , Hierarchical Learning with Off-Policy Correction ( Nachum et al. , 2018 ) tries to support off-policy learning even though that all layers are constantly evolving by correcting the goals present in the transitions using a heuristic method . While goals produced by feudal networks ( Vezhnevets et al. , 2017 ) might be more effective for training , they do not fit our interpretability objectives either since the goal space is a latent space that is not directly understandable to researchers and non-experts . 3 GENERALIZED HINDSIGHT ACTOR-CRITIC WITH TEACHER . Our work builds upon the Hindsight Actor-Critic ( Levy et al. , 2019 ) or HAC , a state-of-the-art algorithm to train hierarchical agents , which achieves excellent performance in some environments beating the hierarchical agent algorithms mentioned before . We refer the interested reader to the original description of HAC ( Levy et al. , 2019 ) due to its non-trivial nature . HAC is designed for a specific setting : environments that provide an end goal and where the only objective is to reach the goal as fast as possible . This specialization leads to 2 limitations : ( 1 ) HAC requires a goal , making it incompatible with all environments which do not provide a goal for the agent and ( 2 ) HAC ignores the rewards given by the environment since it uses an internal reward scheme . This makes it inapplicable to most environments , in which rewards can be given anytime . To address these issues we generalize HAC , creating the HAC-General with Teacher algorithm which doesn ’ t require a goal and that considers the reward given by the environment . To avoid requiring a goal , the policy at the top of the hierarchy produces its output ( a shorter-term goal ) using only the state as input ( no end-goal in the input ) . To take into account the rewards , the objective of the goal-picking policy becomes picking goals such that the maximum amount of reward is collected during the episode . The objective of policy at the bottom of the hierarchy ( the goal-reaching policy ) stays the same : reaching the short-term goal in at most H steps , ignoring environment rewards . 3.1 MAINTAINING THE OPTIMALITY ILLUSION TO ADDRESS NON-STATIONARITY . These changes address the 2 limitations of HAC , but they break HAC ’ s technique to make training effective : addressing the non-stationarity problem , i.e . the problem that since all policies in the hierarchy train in parallel , each policy needs to continuously adapt to the changes in the policies below it in the hierarchy ( whose behavior it relies on ) , which makes training difficult and unstable . The insight of HAC is that if each policy trained under the illusion that all the policies below it were stationary , then it would train faster and more efficiently . Since optimal policies are stationary , HAC attempts to give each policy the illusion that the policy below it is optimal and thus stationary . HAC carefully constructs 3 types of transitions to create this illusion , where a transition is a tuple of the form ( state , action , reward , next state , goal , discount ) . While the 3 types of transitions are detailed in Appendix A for space reasons , we summarize how HAC creates the illusion that the policy below is optimal and how HAC-General With Teacher preserves that illusion . We define some terminology : let π be the policy in the hierarchy for which we create the illusion . We call π the goal-picking policy since it produces goals and call the policy below it in the hierarchy the goal-reaching policy πbelow , since it attempts to reach the goals it receives from π. HAC . In HAC it is simple to give to π the illusion that πbelow is optimal because rewards are only given when the goal-state is reached . As shown in Figure 2 , if the action of π is to pick the goal g and the goal-reaching policy πbelow fails to reach it and reaches state s instead , π ’ s action is replaced by the hindsight action s. The policy πbelow now appears optimal since π picked goal s and πbelow reached it . Problem . This technique breaks down when environment rewards matter and must be maximized , i.e . it breaks down for HAC-General with Teacher . Replacing g by s is not enough anymore to give the illusion that the goal-reaching πbelow acted optimally : while πbelow reached state s , it might not have collected the maximum amount of reward possible . In other words , there might be an alternative path to the same final state where more reward would have been collected ( Figure 3 ) . Since in most environments , it is impractical or impossible to determine if the optimal path was taken ( or what the optimal path is ) , we can not guarantee that πbelow appears optimal . Solution . To address this issue , HAC-General uses a new definition of goals . The new goals have 2 components : a state s which must be reached and a minimum amount of reward rmin which must be collected . As shown in Figure 4 , if the original action/goal is ( s , rmin ) but the goal-reaching policy reaches instead s′ and collects r′ reward , then the goal-picking policy ’ s action is replaced by the hindsight action ( s′ , rhindsight ) where rhindsight ≤ r , creating again the optimality illusion . It is important to note HAC-General creates the same 3 types of transitions as HAC ; the major change is the way goals are defined . Advantages . The goal-picking policy now has 2 mechanisms to maximize the reward it collects1 : ( 1 ) pick goal states s that lead to high rewards and ( 2 ) force the goal-reaching policy to take a highreward path to s by making the minimum reward threshold rmin as high as possible . The second point makes it possible to achieve high reward in RL environments where the reward is also tied to the action , not just the state , since the goal-reaching policy will learn to pick actions that lead to the goal-state but also lead to high reward .

Summary: This paper proposes a hierarchical RL method where the high level controller produces a series of sub-goals in an open-loop fashion which the low-level controller attempts to reach sequentially with the aim of maximising task rewards. The agent is trained using an extension of Hindsight Actor-Critic (HAC) algorithm. The algorithm also leverages a model-free flat policy trained on task rewards as an expert. The approach is evaluated on two tasks: Mountain Car and Lunar Lander.