Year: 2021

Abstract:
We present a two-step hybrid reinforcement learning (RL) policy that is designed to generate interpretable and robust hierarchical policies on the RL problem with graph-based input. Unlike prior deep reinforcement learning policies parameterized by an end-to-end black-box graph neural network, our approach disentangles the decision-making process into two steps. The first step is a simplified classification problem that maps the graph input to an action group where all actions share a similar semantic meaning. The second step implements a sophisticated rule-miner that conducts explicit one-hop reasoning over the graph and identifies decisive edges in the graph input without the necessity of heavy domain knowledge. This two-step hybrid policy presents human-friendly interpretations and achieves better performance in terms of generalization and robustness. Extensive experimental studies on four levels of complex text-based games have demonstrated the superiority of the proposed method compared to the state-of-the-art.

heading: 1 INTRODUCTION
text: Recent years have witnessed the rapid developments of deep reinforcement learning across various domains such as mastering board games (Silver et al., 2016; Schrittwieser et al., 2020) playing video games (Mnih et al., 2015), and chip design (Mirhoseini et al., 2021), etc. The larger and complicated architecture of deep reinforcement learning models empowered the capabilities of resolving challenging tasks while brought in significant challenges of interpreting the decision making process of those complex policies (Puiutta & Veith, 2020). This trade-off between performance and interpretability becomes an inevitable issue when DRL is applied to high stakes applications such as health care (Rudin, 2019). In this work, we focus on graph-based interpretable reinforcement learning (Zambaldi et al., 2018; Waradpande et al., 2020) as the graph representation is expressive in various domains including drug discovery (Patel et al., 2020), visual question answering (Hildebrandt et al., 2020), and embodied AI (Chaplot et al., 2020; Huang et al., 2019), etc. Another benefit of studying graph-based RL is that the graph structure can provide natural explanations of the decisionmaking process without the necessity of introducing new programms (Verma et al., 2018) or heavy domain knowledge (Bastani et al., 2018) for interpretation. Prior works in interpretable reinforcement learning (Verma et al., 2018; Madumal et al., 2020; Liu et al., 2018) either works on restricted policy class (Liu et al., 2018) that leads to downgraded performance, or the interpretablility (Shu et al., 2017; Zambaldi et al., 2018) is limited. Another common issue of interpretable RL is that the provided explanation is generally difficult to comprehend for non-experts (Du et al., 2019).
To resolve the challenges mentioned above, we propose a novel two-step hybrid decision-making process for general deep reinforcement learning methods with graph input. Our approach is inspired by the observation of human decision-making. When confront complicated tasks that involve expertise from multiple domains, we typically identify which domain of expert we would like to consult first and then search for specific knowledge or solutions to solve the problem. Recognizing the scope of the problem significantly reduces the search space of downstream tasks, which leads to a more simplified problem compared to find the exact solution in all domains directly. As an analogy of this procedure, we disentangle a complicated deep reinforcement learning policy into a classification the problem for problem type selection and rule-miner. The classification establishes a mapping from complex graph input into an action type, which handles high-order logical interactions among node and edge representations with graph neural network. The rule miner conducts explicit one-hop reasoning over the graph and provides user-friendly selective explanations (Du et al., 2019) by mining several
decisive edges. This two-step decision making is essential not only for providing interpretability, but also for generalization and robustness. It is intuitive to see that the simplified classification is easier to achieve better generalization and robustness than the original complicated RL policy. Furthermore, the rule-miner identifying key edges in the graph is much more robust to the noisy perturbations on the irrelevant graph components.
In summary, our contributions are three folds:
• We formalize an interpretable deep reinforcement learning problem based on graph input. • We propose a two-step decision-making framework that achieves far better performance in
terms of generalization and robustness and provides human-friendly interpretations. • Extensive experiments on several text-based games (Côté et al., 2018) demonstrated that the
proposed approach achieves a new state-of-the-art performance for both generalization and robustness.

heading: 2 RELATED WORKS

heading: 2.1 INTERPRETABLE RL
text: Instead of using deep RL as a black box, researchers also worked on making deep RL more interpertable. (Mott et al., 2019; Zambaldi et al., 2018) introduce some sorts of attention mechanisms into policy networks and explain decision making process by analyzing attention weights. (Verma et al., 2018) combines program synthesis and PIDs in classic control to solve continuous control problems, while keep the decision making interpretable by explicitly writing the program. Combining symbolic planning and deep RL has similar effects, like (Lyu et al., 2019). Tree-based policies (Bastani et al., 2018) are favorable in interpretable RL, since they are more human-readable and easy to verify. However, many existing works of interpretable RL sacrifices the performance for the interpretability. In contrast, our method achieves even better generalization and robustness while providing interpretability.

heading: 2.2 HIERARCHICAL RL
text: Hierarchical RL focuses on decomposing a long-horizon tasks into several sub-tasks, and applying a policy with hierarchical structure to solve the task. (Vezhnevets et al., 2017; Frans et al., 2018) try to solve generic long-horizon tasks with a two-level policy, where a high-level policy will generate a latent sub-goal for low-level policies, or directly select a low-level policy, and the low-level policy is responsible for generating the final action. In some robot locomotion tasks, people usually manually define the choices of sub-goals to boost the performance and lower the learning difficulty, like (Nachum et al., 2018; Levy et al., 2018). Our two-step hybrid policy is reminiscent of the two-level policy architecture in hierarchical RL. However, the high-level policy in hiereachical RL is mainly used to generate a sub-goal while the action pruner in our hybrid policy is used to reduce the number of action candidates. And the motivations of our method and hierarchical RL are also different: hierarchical RL targets at decomposing a long-horizon tasks into several pieces, but our method focuses on interpertability, generalizability and robustness.

heading: 2.3 REFACTORING POLICY
text: Given a trained neural network-based policies, one may want to refactor the policy into another architecture, so as to improve the generalizibility or interpretablity. And this refactoring processs is usually done by imitation learning. The desired architectures of the new policy can be decision tree (Bastani et al., 2018), symbolic policy (Landajuela et al., 2021), a mixture of program and neural networks (Sun et al., 2018), or graph neural networks (Mu et al., 2020). In our method, we also use similar techniques to refactor a reinforcement learning policy into a two-step hybrid policy.

heading: 2.4 COMBINE NEURAL NETWORKS WITH RULE-BASED MODELS
text: Neural networks and rule-based models excel at different aspects, and many works have explored how to bring them together, like (Chiu et al., 1997; Goodman et al., 1990; Ray & Chakrabarti, 2020;
Okajima & Sadamasa, 2019; Greenspan et al., 1992). Therefore various ways to combine neural networks with rule-based models. For example, (Wang, 2019) combines them in a ”horizontal” way, i.e., some of the data are assigned to rule-based models while the others are handled by neural networks. In contrast, our work combines neural networks and rule-based models in a ”vertical” way, i.e., the two models collaborate to make decisions in a two-step manner.

heading: 3 A GENERAL FRAMEWORK FOR TWO-STEP HYBRID DECISION MAKING
text: In this section, we first describe our problem setting including key assumptions and a general framework that formulates the decision-making process in a two-step manner.
We consider a discrete time Markov Decision Process (MDP) with a discrete state space S and the action spaceA is finite. The environment dynamics is denoted as P = {p(s′|s, a),∀s, s′ ∈ S, a ∈ A}. Given an action set A and the current state st ∈ S, our goal is to learn a policy (π) that select an action at ∈ A to maximize the long-term reward Eπ[ ∑T i=1 r(si, ai)]. Assume we are able to group the actions into several mutual exclusive action types (Ak) according to its semantic meanings. More concretely, the k-th action type Ak = {a1k, ..., ank} denotes a subset of actions in original action space Ak ⊆ A. Then we have A1, A2, .., AK ⊆ A, Ai ∩ Aj = ∅(i 6= j),∪Ki=1Ai = A. It is worth noting that the number of action type K is usually more than an order of magnitude smaller than original actions (K |A|). Let the policy π = 〈fp, fs〉 represented by a hybrid model that consists of action pruner fp and action selector fs. The action pruner is used to prune all available action candidates to a single action type, i.e., fp(si) = k, where k is the index of chosen action type and Ak ∈ {A1, A2, .., AK}. Then the action selector is used to select a specific action given the action type chosen by the action pruner, i.e., fs(si, Ak) = ai, where ai ∈ Ak and k = fp(si). Intuitively, this design intends to disentangle different phases in decision-making process to two different modules. On one hand, determining the action type typically involves high-order relationships and needs a model with strong expressive power, then neural network is a good candidate in this regard. On the other hand, selecting an action within a specific action type can resort to rule-based approaches, which is essential in providing strong intrepretability, generalizability and robustness.
Figure 1 shows the overall pipeline of our two-step hybrid decision making. The agent will receive a state si at each time step i. At time step i, we will first call the action pruner fp(si) to select the action type Ai. Then the rule-based action selector fs(si, Ai) will take as inputs the current state si and the action type Ai given by action pruner to select the specific action to be executed in this step.

heading: 4 TWO-STEP HYBRID POLICY FOR TEXT-BASED GAMES WITH GRAPH INPUT

heading: 4.1 OVERVIEW
text: In this section, we instantiate our proposed framework in the setting of text-based games (Côté et al., 2018) with graph input. In text-based games, the agent receives a knowledge graph (as shown in Figure 3) that describes the current state of the environment and the task to be completed, e.g., the task can be represented by several edges like (“potato”, “needs”, “diced”). Our goal is to learn a policy that maps the input knowledge graph to an action from the provided action set A. Each action aj ∈ A is a short sentence, e.g., “take apple from fridge”. In this setting, we use graph neural networks (GNNs) as the action pruner fp, and a rule-based model as the action selector fs. We will elaborate the details of the action pruner and the action selector in Sec 4.3 and Sec 4.4, respectively.
Training the GNN-based action pruner and the rule-based action selector by reinforcement learning is nontrivial since the whole pipeline is not end-to-end differential. Therefore, we propose to learn both models separately from a demonstration dataset, and the demonstration dataset can be obtained by a trained reinforcement learning agent. We will elaborate the details in Sec 4.2.1. This process is inspired by existing works like (Bastani et al., 2018; Landajuela et al., 2021; Sun et al., 2018; Mu et al., 2020), where policies trained by reinforcement learning can be refactorized into other forms.

heading: 4.2 PREPARING DATASETS FOR LEARNING TWO-STEP HYBRID POLICY

heading: 4.2.1 DEMONSTRATION ACQUISITION
text: Given a set of interactive training environments, the goal of this step is to obtain a demonstration dataset, which contains state-action pairs from a policy achieving high reward in the training environment. Specifically, we target at generating a demonstration dataset D = {(si, π(si))}Ni=1, where si is the input state (a knowledge graph in our cases) from the training environments, and π(si) is the output of demonstration policy on the state si. The representation of π(si) is flexible: it can be an action, the logits, or any latent representation, which indicates the demonstration action distribution. This dataset will be used for learning our two-step hybrid policy.
We refer the policy used to generate demonstration as teacher policy. It can be obtained in any way, as long as it provides reasonable and good supervision on how to solve the task in the training environments, and it is not necessary to have strong interpretablity, generaliziabilty or robustness. Therefore, deep reinforcement learning is a great tool to learn such a policy.
After obtained the teacher policy, we use it to interact with the training environments to collect states and label them with the output of the teacher policy. During the interaction, we add action noise to perturb the state distribution in demonstration dataset. A more diversified state distribution is beneficial to the performance of our two-step hybrid approach.

heading: 4.2.2 ACTION GROUPING AND DEMONSTRATION SPLITS
text: As mentioned above, the action pruner is responsible for selecting the correct type of actions for the current state, so that the action candidates will be pruned to a smaller set. Then, a rule-based model will work on this specific action type to select the action to be executed. Therefore, we needs to label each state with the action type. Since the action for each state is already in the dataset, we just need to convert each action into an action type. In other words, we need to group all possible actions into several action types.
In text-based games, each action is a short sentence with semantic meaning, e.g., “take apple from fridge”. Intuitively, we can group the actions by their semantic meaning. For example, if the actions are “take apple from fridge”, “slice potato with knife”, “dice cheese with knife”, “cook carrot with oven”, and “cook onion with stove”, we can group them into three types: “take” actions, “cut” actions, and “cook” actions. And we make sure all the actions with the same action type follow the same template, e.g., the template of “take” action can be written as “take object from receptacle”, where object and receptacle can be instantiated by appropriate words.
Grouping actions by semantic meanings also aligns with our motivation: 1) After we obtained K group of actions, we substantially alleviate the workload of neural networks from learning a policy that maps states to an action in a varying size and high-dimensional action space to a simplified classification with a much smaller number of classification categories. This simplification could greatly reduces the sample-complexity. 2) Furthermore, this step also eliminates the search space to enable a feasible candidates set for the rule-miner.
Formally, given a dataset D = {(si, π(si))}Ni=1, we will generate an action type for each state, policy output (si, π(si)) pair. The action type ki is determined by ki = h(π(si)), where h(π(si)) is a domain-specific function which takes the semantic meaning of the action into consideration. As a result of the clustering process, we will get a new demonstration dataset with action types D = {(si, π(si), ki)}Ni=1. In addition, we can also split the demonstration dataset based on the action types to get K subset of the original demonstration dataset, {D1,D2, ...,DK}, where Dk = {si, π(si)|h(π(si)) = k}.

heading: 4.3 GNN-BASED ACTION PRUNER
text: The action pruner needs to output an action type based on the input knowledge graph, so it is essentially a classifier. Given the demonstration dataset D = {(si, ki)}Ni=1 obtained in the last step, we want to train a classifier fp(s; θ) = k, where k ∈ {1, 2, ...,K} is an action type. This is a conventional classification problem which can be solved by minimizing cross entropy loss:
θ = argmin θ − ∑ i K∑ j=1 kji log(f j θ (si)),
where fθ(si) outputs a probability distribution over the K action types, f j θ (si) denotes the j-th action type’s probability. kji ∈ {0, 1} denotes denotes whether the action type j was chosen in the demonstration dataset at state i.

heading: 4.4 RULE-BASED ACTION SELECTOR

heading: 4.4.1 ABSTRACT SUPPORTING EDGE SETS
text: When the input is a knowledge graph, the action is naturally strongly correlated with some critical edges. For example, (“potato”, “needs”, “diced”) and (“potato”, “in”, “player”) can lead to the action “dice potato”. We refer those decisive edges correlating to an action as the supporting edge set of this action. Since we have grouped actions by their semantic meanings, actions within each action type are actually supported by similar edges. For example, “dice potato” is supported by (“potato”, “in”, “player”), and “dice apple” is supported by (“apple”, “in”, “player”). As mentioned in Sec 4.2.2, each action type comes with an action template like “dice object”. Based on the action template, we can perform some sorts of abstraction. For example, given an input knowledge graph labeled with action “dice apple”, we can replace all the “apple” appearing in the graph edges and the action with an abstract name “object”. Then we can say the action “dice object” is essentially supported by (“object”, “in”, “player”), where the two “object” should be instantiated by the same word. Under this kind of abstraction, different actions within the same action type can share a same abstract supporting edge set which contains edges with abstract names.
The abstract supporting edge set indicates the decisive edges for an action type, and it can be instantiated for each specific action. For example, to check whether the action “dice apple” should be executed, the abstract edge (“object”, “in”, “player”) will be instantiated to edge (“apple”, “in”, “player”). Then, the existence of (“apple”, “in”, “player”) in input knowledge graph becomes an evidence for selecting the action “dice apple”. We aim at finding an abstract supporting edge set for each action type, and it will be used during inference.

heading: 4.4.2 MINE ABSTRACT SUPPORTING EDGE SETS FROM DEMONSTRATIONS
text: Finding the abstract supporting edge set for each action type is actually a rule mining process. There are several off-the-shelf rule miners like FP-Growth (Han et al., 2004), Apriori (Agrawal et al., 1994), Eclat (Zaki, 2000), etc. , but they are not designed for knowledge graphs. Thus, we propose a simple yet effective rule miner for our setting to discover the supporting edge sets.
To find the Abstract Supporting Edge set ASE(Ak) for each action type Ak, we designed a numerical statistic that is intended to reflect the importance of an edge when taking an action a ∈ A, and this numerical statistic is inspired by tf-idf (Rajaraman & Ullman, 2011). Formally, for action type Ak, we have a subset of demonstration dataset Dk = {si} (ignoring πi here). Under the abstraction mentioned above, we can count the edge frequency for every (abstract) edge e in Dk:
freqk(e) = |{si|si ∈ Dk, e ∈ si}|
|Dk| . Similarly, we can also count the edge frequency for the entire demonstration dataset:
freq(e) = |{si|si ∈ D, e ∈ si}|
|D| . And we can define an importance score of an edge w.r.t to the action type Ak:
Ik(e) = freqk(e) · log( 1
freq(e) ),
where the term freqk(e) is similar to the term-frequency (tf), and the term log( 1freq(e) ) is similar to the inverse document frequency (idf).
Then we can get the ASE(Ak) by selecting the edges with the importance higher than a threshold, i.e., ASE(Ak) = {e|Ia(e) > τ}, where τ is a hyperparameter shared across all action types.

heading: 4.4.3 INFERENCE BASED ON SUPPORTING EDGES
text: During inference, we use the supporting edge sets to score each action within the action type provided by action pruner, and select the action with the highest score. Figure 2 shows a concrete example of using supporting edge set to score an action.
Firstly, the action pruner outputs an action type based on the input knowledge graph, and we can retrieve the abstract supporting edge set of this action type. Secondly, given an action within the action type, we can instantiate the abstract supporting edge set to a specific supporting edge set by replacing the abstract names by the concrete words according to the action, e.g, (“object”, “in”, “player”) will be instantiated to (“potato”, “in”, “player”) if the action is “cook potato with oven”. Finally, we can compare the input knowledge graph with the supporting edge set of each action to figure out which supporting edge set is best covered by the input knowledge graph. The number of overlapped edges between the supporting edge set and the the input knowledge graph will be regarded as the score of the action.
Formally, the inference process of our rule-based action selector can be described as follows
fs(s,A) = argmax a∈A
|s ∩ SE(a)|,
where SE(a) is the supporting edge set associated with action a, which is obtained by instantiating the abstract supporting edge set of action type A.

heading: 5 EXPERIMENTS

heading: 5.1 DATASET SETUP
text: We evaluate our method on TextWorld, which is a framework for designing text-based interactive games. More specifically, we use the TextWorld games generated by GATA (Adhikari et al., 2020). In these games, the agent is asked to cook a meal according to given recipes. It requires the agent to navigate among different rooms to locate and collect food ingredients specified in the recipe, process the food ingredients appropriately, and finally cook and eat a meal.
The state received by the agent is a knowledge graph describing all the necessary information about the game. All the nodes and edges in the knowledge graph are represented in text. Figure 3 shows a partial example of input knowledge graph. The actions are also represented in text. Note that the number of available actions vary from state to state, so most of of existing network architecture used deep reinforcement learning cannot be directly used here.
The games have four different difficulty levels, and each difficulty level contains 20 training, 20 validation, and 20 test environments, which are sampled from a distribution based on the difficulty level. The higher the difficulty levels are, the more complicated recipe will be and the more rooms food ingredients will be distributed among. Statistics of the games are shown in Table 1. For evaluating model generalizability, we select the top-performing agent on validation sets and report its test scores; all validation and test games are unseen in the training set. 1

heading: 5.2 IMPLEMENTATION DETAILS

heading: 5.2.1 PROCESS KNOWLEDGE GRAPHS BY GNNS
text: Graph neural networks (GNNs) serve as backbone networks in both our RL teacher policy and the action pruner in our two-step hybrid policy. More specifically, we use Relational-GCN to take edge attributes into consideration.
The relations and nodes of the input knowledge graphs are initially represented in text (words or short sentences). We use fastText (Mikolov et al., 2017), pre-trained on Common Crawl, to convert each token to an embedding, and then average all the token embeddings to obtain the mean fastText embedding of the entire text string. These mean fastText embeddings are pre-computed and fixed during training. For each relation and node, we append their corresponding mean fastText embedding with a trainable embedding vector. The concatenated embeddings are then used as the numerical representation of each relation and node, which are further passed through GNNs to get the latent embedding of the entire input knowledge graph.

heading: 5.2.2 DEMONSTRATION ACQUISITION
text: To collect demonstration dataset, we first train a teacher policy by DQN (Mnih et al., 2015) in the training environments, which can converge to a near-optimal solution. To adopt variable number of available actions, we let the policy network take the state and an available action as input and output a score for this action. And we can select the action with maximum score as the final output.
The trained teacher policy is used to collect 300K samples through the interaction with the environment, and label them with the taken actions, as illustrated in Sec 4.2.1. When collecting the demonstration dataset, we use -greedy exploration strategy to increase the diversity of states.
1The code to reproduce all results in this work will be publicly available.

heading: 5.3 RESULTS

heading: 5.3.1 INTERPRETABILITY
text: The interpretablity of our two-step policy is two-fold: 1) the transparent two-step decision making process; 2) the rule-based models make decisions in a way which is easy to interpret by human.
Table 2 shows some representative rules discovered by our rule miners. We observed that all of the four discovered abstract supporting edges are indeed prerequisites of the “cut” actions. In particular, the abstract supporting edge (“object”, “needs”, “verb passive”) is a crucial prerequisite for the agent to select the correct verb in the “cut” actions. For example, if the input knowledge graph contains the edge (“potato”, “needs”, “sliced”), then the action “slice potato with knife” will get one more score than others because this edge is in the supporting edge set of action “slice potato with knife”.
Since we clearly see the rule-based model makes decisions based on these rules, it is not hard to check whether this model works in a correct way. In this way, the agent can certainly select the correct “cut” action even in unseen test environments.

heading: 5.3.2 GENERALIZATION
text: We compare with two baselines: GATA-GTF and vanilla RL. GATA-GTF is a variant from GATA (Adhikari et al., 2020), and it uses the same ground-truth graph input as us. GATA-GTF processes the input graphs by relational graph convolutional networks (R-GCNs) (Schlichtkrull et al., 2018), and the whole pipeline is trained by DQN (Mnih et al., 2015). Vanilla RL is essentially the same method with GATA-GTF, but implemented by ourselves. With better implementation and careful hyperparameter tuning, it performs better than the implementation released by the GATA authors. And vanilla RL also serves the teacher policy which used to generate demonstrations for our method.
Table 3 shows the normalized scores of different methods on both training environments and test environment in TextWorld. The result of GATA-GTF is obtained from its paper (Adhikari et al., 2020). In all the environments, our agent achieves better generalization performance the vanilla RL (which is our RL teacher) and GATA-GTF baselines. Our agent can generalize pretty well to the
unseen test environments in all difficulty levels, while the performance of GATA-GTF and vanilla RL performs poorly in unseen test environments.

heading: 5.3.3 ROBUSTNESS
text: As mentioned above, our method also aims at robustness. To evaluate the robustness of our models, we add different levels of noises to input knowledge graphs and regard the performance of the agents under noisy inputs. In this paper, we define the noise on a knowledge graph as adding additional edges to the graph or dropping existing edges in the graph. Formally, we add noise at (k, p)-level to input knowledge graph in the following way:
1. Add edges: dk ∗ |E|e additional edges will be randomly generated and added to the graph, where E is the edge set of the graph. For each edge hi, ti, ri, head node hi and tail node ti are sampled from Vall and relation ri is sampled from Rall, where Vall means the set of all possible nodes and Rall is the set of all possible relations.
2. Drop edges: dp ∗ |E|e edges will be dropped from the graph. Note that only original edges will be dropped, and the randomly added edges will not be dropped.
Table 4 shows the performance of vanilla RL and our method under noisy input graphs generated in the above mentioned way. Note that we test robustness in training environments instead of test environments, because the performance of both agents in test environments are not good enough and examining robustness of poor-performed agents does not make too much sense.
We observed that in difficulty level 1, both agents are very robust under the noisy inputs. However, in difficulty level 2 & 3 & 4, the performance of the RL agents are hurts a lot by the input noises, especially in the difficulty level 4. In contrast, the agents obtained by our method are still performs pretty good and the performance drops significantly less than the RL agents.

heading: 6 CONCLUSION
text: In this work, we propose a two-step hybrid policy for graph-based reinforcement learning. The two-step hybrid policy disentangles complicated black-box deep reinforcement learning policy into an action pruner and an action selector. The joint effort of two modules can not only generate human-friendly explanations of each decision, but also provide significantly better performance in terms of generalization and robustness. We conduct comprehensive experiments on the representative text-based environments (Côté et al., 2018) to demonstrate the effectiveness of our approach.

Review summaries:
This work targets to the problem of limited interpretability in RL. To solve the problem, the authors propose a two-step hybrid model, including action pruner and action selector. The proposed method is evaluated on the text games, with results showing its ability of interpretation, generalization, and robustness.

This paper proposes a pipeline to determine (primitive) actions to be performed in states described by knowledge graphs. The pipeline is divided into two steps. The first step, Action Pruner, maps states into abstract actions (i.e., clusters of primitive actions) and is performed by a trained GNN with data from a teacher. The second step, Action Selector, is a rule-based system that defines the primitive action to be performed among those included in the abstract action given by the previous step. The Action Selector defines the primitive action based on the GNN Abstract Supporting Edge set (ASE), also defined from the training data provided by the teacher. The proposal presented good results in games with text-based environments and also allows the generated decisions to be human-friendly explanations.

This paper is about improving the interpretability of agents trained on reinforcement learning problems where the input observation is given as a graph. Specifically, the authors propose to split the decision-making process in two: 1) an action pruner, and 2) an action selector. Experiments were conducted on a suite of TextWorld environments that have their state exposed as a knowledge graph. Empirically, it was shown that the proposed two-step approach is better at generalizing on unseen environments and is more robust to noisy perturbations on the input graphs.

This paper introduces a two-step method for RL problems with graph-based inputs. This method trains a teacher policy which it then uses to construct a policy "pruner" which predicts the kind of action needed. It then uses a ideas from retrieval to select the edges needed to complete the action.