Learning Two-Step Hybrid Policy for Graph-Based Interpretable Reinforcement Learning
Authors: Tongzhou Mu
Kaixiang Lin
Feiyang Niu
Govind Thattai
Venue: ICLR 2022 Submitted (2022)
Keywords: Interpretable Reinforcement Learning
Generalization
Robustness

Abstract: We present a two-step hybrid reinforcement learning (RL) policy that is designed to generate interpretable and robust hierarchical policies
on the RL problem with graph-based input. Unlike prior deep reinforcement learning policies parameterized by an end-to-end black-box graph neural network, our approach disentangles the decision-making process into two steps. The first step is a simplified classification problem that maps the graph input to an action group where all actions share a similar semantic meaning. The second step implements a sophisticated rule-miner that conduct explicit one-hop reasoning over the graph and identifies decisive edges in the graph input without the necessity of heavy domain knowledge. This two-step hybrid policy presents human-friendly interpretations and achieves better performance in terms of generalization and robustness. Extensive experimental studies on four levels of complex text-based games have demonstrated the superiority of the proposed method compared to the state-of-the-art. 

Under review as a conference paper at ICLR 2022
LEARNING TWO-STEP HYBRID POLICY FOR GRAPH- BASED INTERPRETABLE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We present a two-step hybrid reinforcement learning (RL) policy that designed to generate interpretable and robust hierarchical policies on the RL problem with graph-based input. Unlike prior deep reinforcement learning policies parameterized by an end-to-end black-box graph neural network, our approach disentangles the decision-making process into two steps. The first step a simplified classification problem that maps the graph input to an action group where all actions share sim- ilar semantic meaning. The second step implements a sophisticated rule-mine milerminer that conducts explicit one-hop reasoning over the graph and identifies decisive edges in the graph input without the necessity of heavy domain knowledge. This two-step hybrid policy presents human- friend-f interpretations and achieves better perfor- mance in terms of generalization and robustness. Extensive experimental studies four levels of complex text-based games have demonstrated the superiority of the proposed method compared to the state-of-the-art
INTRODUCTION
Recent years have witnessed the rapid developments of deep reinforcement learning across various domains such as mastering board games (Silver et al., 2016; Schrittwieser et al. 2020) playing video games (Mnih et al., 2015), and chip design (Mirhoseini et al. [001]) etc. The larger and complicated architecture of deep reinforcement learning models empowered the capabilities of resolving challenging tasks while brought in significant challenges of interpreting the decision making process of those complex policies (Puiutta & Veith 2020). This trade-off between performance and interpretability becomes an inevitable issue when DRL is applied to high stakes applications such as health care (Rudin) 2019). In this work, we focus on graph-based
interpretable reinforcement learning (Zambaldi et 20118 Waradpande et al.] 2020) as the graph representation is expressive in various domains including drug discovery (Patel et al. 2020). visual question answering (Hildebrandi
et al. 2020), and embodied AI (Chaplot al. 2020; Huang et al. 2019), etc. Another benefit of studying graph-based RL is that the graph structure can provide natural explanations the decision- making process without the necessity of introducing new programms (Verma et al.) 2018) heavy domain knowledge (Bastani et al.] (008)) for interpretation. Prior works in interpretable reinforcement learning (Verma et al. 2018 Madumal et al. 2020 Liu et al. 2018) either works restricted policy class (Liu et al. 2018) that leads to downgraded performance, or the interpretablility (Shu et al. [017]] Zambaldi et 2018) is limited. Another common issue of interpretable RL is that the provided explanation is generally difficult to comprehend for non-experts (Du et al. 2019).
To resolve the challenges mentioned above, we propose novel two-step hybrid decision-making process for general deep reinforcement learning methods with graph input. Our approach is inspired by the observation of human decision-making When confront complicated tasks that involve expertise from multiple domains, we typically identify which domain of expert we would like to consult first and then search for specific knowledge or solutions to solve the problem. Recognizing the scope of the problem significantly reduces the search space of downstream tasks, which leads to a more simplified problem compared to find the exact solution all domains directly. As an analogy of this procedure, we disentangle complicated deep reinforcement learning policy into a classification the problem for problem type selection and rule-miner. The classification establishes a mapping from complex graph input into action type, which handles high-order logical interactions among node and edge representations with graph neural network. The rule miner conducts explicit one-hop reasoning over the graph and provides user-friee feienfiienly selective explanations (Du et al. 2019) by mining several

---

Under review as a conference paper at ICLR 2022
decisive edges. This two-step decision making is essential not only for providing interpretability. but also for generalization and robustness. It is intuitive to see that the simplified classification is easier to achieve better generalization and robustness than the original complicated RL policy. Furthermore, the rule-miner identifying key edges in the graph is much more robust to the noisy perturbations on the irrelevant graph components.
In summary, our contributions are three folds:
We formalize an interpretable deep reinforcement learning problem based on graph input.
We propose two-step Decision-making framework that achieves far better performance in terms generalization and robustness and provides human human-friendie friendly interpretations.
Extensive experiments on several text-based games (Côté et al., 2018) demonstrated that the proposed approach achieves new state-of-the-art performance for both generalization and robustness.
RELATED WORKS
2.1 INTERPRETABLE RL
Instead of using deep RL black box, researchers also worked on making deep RL more inter- pertable. (Mott et al.] 2019 Zambaldi et al., (2018) introduce some sorts of attention mechanisms into policy networks and explain decision making process analyzing attention weights. (Vermal et al. 2018) combines program synthesis and PIDs in classic control to solve continuous control problems, while keep the decision making interpretable by explicitly writing the program. Combining symbolic planning and deep RL has similar effects, like (Lyu et al. 2019). Tree-based policies (Bastani af., 2018) are favorable in interpretable RL, since they are more human-readable and easy to verify. However, many existing works of interpretable RL sacrifices the performance for the interpretability. In contrast, our method achieves even better generalization and robustness while providing interpretability.
2.2 HIERARCHICAL RL
Hierarchical RL focuses on decomposing a long-horizon tasks into several sub-tasks, and applying a policy with hierarchical structure solve the task. (Vezhnevets et al., [017]] Frans et al. 2018) try to solve generic long-horizon tasks with a two-level policy, where a high-level policy will generate a latent sub-goal for low-level policies, or directly select low-level policy, and the low-level policy is responsible for generating the final action. In some robot locomotion tasks, people usually manually define the choices of sub-goals boost the performance and lower the learning difficulty, like (Nachum et al. 2018 Levy et al. 2018). Our two-step hybrid policy is reminiscent of the two-level hierarchical RL. However, the high-level policy in hiereachical RL mainly
policy architecture used to generate sub-goal while the action pruner in our hybrid policy is used to reduce the number of action candidates. And the motivations of our method and hierarchical RL are also different: hierarchical RL targets at decomposing long-horizon tasks into several pieces, but our method focuses on interpertability, generalizability and robustness.
2.3 REFACTORING POLICY
Given a trained neural network-based policies, one may want refactor the policy into another architecture, so as to improve the generalizibility or interpretablity. And this refactoring processs is usually done by imitation learning. The desired architectures of the new policy can be decision tree (Bastani et al., 2018), symbolic policy (Landajuela et al.) (22I)) ), mixture of program and neural networks (Sun et al. 2018), or graph neural networks (Mu al. 2020). In our method, we also use similar techniques to refactor reinforcement learning policy into two-step hybrid policy.
2.4 COMBINE NEURAL NETWORKS WITH RULE-BASED MODELS
Neural networks and rule-based models excel at different aspects, and many works have explored how to bring them together, like (Chiu al. 1997 Goodman et al. 1990 Ray & Chakrabarti 2020
2

---

Under review as a conference paper at ICLR 2022
Okajima & Sadamasa 2019 Greenspan al. 1992). Therefore various ways to combine neural networks with rule-based models. For example, (Wang 2019) combines them in a "horizontal" way, i.e., some of the data assigned to rule-based models while the others handled by neural networks. In contrast, our work combines neural networks and rule-based models in a "vertical" way, i.e., the two models collaborate to make decisions in two-step manner.
3 A GENERAL FRAMEWORK FOR TWO-STEP HYBRID DECISION MAKING
In this section, we first describe our problem setting including key assumptions and a general framework that formulates the decision-making process in a two-step manner.
We consider a discrete time Markov Decision Process (MDP) with a discrete state space S and the action space A finite. The environment dynamics is denoted as P {p(s'|s, a), Vs, s' E S, a E A]. Given an action set A and the current state 8, E S, our goal is to learn a policy (I)) that select an action de E A to maximize the long-term reward E,, 1> Li=1 r(s;, a.)] Assume we are able to group the actions into several mutual exclusive action types (A&) according to its semantic meanings. More concretely, the k-th action type A₃- .a......
-
...; d²) denotes a subset of actions in original action space A& S A. Then we have A1 A2, .., Ax A, A, A, Mi + j), / A4 = A. It is worth noting that the number of action type K is usually more than an order of magnitude smaller than original actions (K < [A))
Let the policy π = (Jp. fa) represented by hybrid model that consists of action pruner fp and action selector fa. The action pruner is used prune all available action candidates to single action type, ie., f5():) = k. where k is the index of chosen action type and Ag E A2,, A2,... Ax). Then the action selector is used select specific action given the action type chosen by the action pruner, /a(5, A+) ag where 0g E A₁ and k- = [p()))
Intuitively, this design intends to disentangle different phases in decision-making process two dif- ferent modules. On one hand, determining the action type typically involves high-order relationships and needs model with strong expressive power, then neural network is good candidate in this regard. On the other hand, selecting an action within specific action type can resort to rule- based approaches, which is essential in providing strong intrepretability, generalizability and robustness.
Figure I shows the overall pipeline of our two-step hybrid decision making. The agent will receive a state 8; at each time step i. time step i, we will first call the action pruner (p((;) to select the
action type A, Then the rule-based action selector [=(8; A,) will take as inputs the current state 8; and the action type A, given by action pruner to select the specific action to be executed in this step.
NN-based Action Pruner
Rule-based
State
Action Type
Action
Action Selector
Figure 1: Overall pipeline of the two-step hybrid decision making.
4 TWO-STEP HYBRID POLICY FOR TEXT-BASED GAMES WITH GRAPH INPUT
4.1 OVERVIEW
In this section, we instantiate our proposed framework in the setting of text-based games (Côté et al. 2018) with graph input. In text-based games, the agent receives knowledge graph (as shown in Figure 3) that describes the current state of the environment and the task to be completed, e.g., the task can be represented by several edges like ("potato", "needs", "diced")), Our goal is to learn a policy that maps the input knowledge graph to action from the provided action set A. Each action aj E A is a short sentence, e.g., "take apple from fridge".
In this setting, we use graph neural networks (GNNs) as the action pruner fp. and a rule-based model as the action selector f.. We will elaborate the details of the action pruner and the action selector in Sec 2.c and Sec 4.4 respectively.
3

---

Under review as a conference paper at ICLR 2022
Training the GNN-based action pruner and the rule-based action selector by reinforcement learning is nontrivial since the whole pipeline is not end-to-end differential. Therefore. we propose to learn both models separately from demonstration dataset, and the demonstration dataset can be obtained by a trained reinforcement learning agent. We will elaborate the details in Sec 4.2. 1 This process is
inspired by existing works like (Bastani al., 2018 Landajuela al. 2021 Sun al., (0188) Mu et al. 2020), where policies trained by reinforcement learning can be refactorized into other forms.
4.2
PREPARING DATASETS FOR LEARNING TWO-STEP HYBRID POLICY
4.2.1
DEMONSTRATION ACQUISITION
Given a set of interactive training environments, the goal of this step is to obtain a demonstration dataset, which contains state-action pairs from a policy achieving high reward in the training environ- ment. Specifically.
target generating demonstration dataset D {(s (s+))) where s; is the input state (a knowledge graph in our cases) from the training environments, and (Ts;) is the output of demonstration policy the state 8;- The representation of Zt (((s)) is flexible: it can be action, the logits, or any latent representation, which indicates the demonstration action distribution. This dataset will be used for learning our two-step hybrid policy.
We refer the policy used to generate demonstration as teacher policy. It can be obtained in any way, long it provides reasonable and good supervision on how to solve the task in the training environments, and it is not necessary to have strong interpretablity, generaliziabilty or robustness. Therefore, deep reinforcement learning is great tool to learn such a policy.
After obtained the teacher policy, we use it to interact with the training environments to collect states and label them with the output of the teacher policy. During the interaction, we add action noise to perturb the state distribution in demonstration dataset. A more diversified state distribution is beneficial to the performance of our two-step hybrid approach.
4.2.2 ACTION GROUPING AND DEMONSTRATION SPLITS
As mentioned above, the action pruner is responsible for selecting the correct type of actions for the current state, so that the action candidates will be pruned to smaller set. Then, rule-based model will work on this specific action type to select the action to be executed. Therefore, we needs to label each state with the action type. Since the action for each state already in the dataset, we just need to convert each action into action type. In other words, we need to group all possible actions into several action types.
In text-based games, each action is short sentence with semantic meaning, e.g., "take apple from fridge". Intuitively, we can group the actions by their semantic meaning. For example, if the actions "take apple from fridge", "slice potato with knife", "dice cheese with knife", "cook carrot with oven", and "cook onion with stove", we can group them into three types: "take" actions, "cut" actions, and "cook" actions. And we make sure all the actions with the same action type follow the same template, e.g., the template of "take" action can be written as "take object from receptacle", where object and receptacle can be instantiated by appropriate words.
Grouping actions by semantic meanings also aligns with our motivation: 1) After we obtained K group of actions, we substantially alleviate the workload of neural networks from learning a policy that maps states to an action in varying size and high- diggn-iinniional action space to simplified classification with a much smaller number of classification categories. This simplification could greatly reduces the sample-complexity 2) Furthermore, this step also eliminates the search space to enable feasible candidates set for the rule-miner.
Formally, given dataset D {si x (s)))) we will generate an action type for each state, policy output (st, ग (s,)) pair. The action type kū is determined by k; = h((T(s)), where h(((ss)) is domain-specific function which takes the semantic meaning of the action into consideration. As a result of the clustering process, we will get a new demonstration dataset with action types D {(s (s:), J, In addition, can also split the demonstration dataset based on the action types to get K subset of the original demonstration dataset, (D,, D2, Dx }. where Dj- = (s,, n (s:) [M(T (=(()))))) k).
4

---

Under review as a conference paper at ICLR 2022
4.3 GNN-BASED ACTION PRUNER
The action pruner needs to oulput an action type based on the input knowledge graph, so it is essentially classifier. Given the demonstration dataset D {ss, kig J== obtained in the last step, we want to train a classifier fp(s;0) 0) = k, where k E (1,2,... -.., 2,2,,.., Kj is an action type. This is a conventional classification problem which can be solved by minimizing cross entropy loss:
K M4
0 arg min
where fo();) outputs a probability distribution over the K action types, (s;) denotes the j-th action type's probability. kę {0, denotes denotes whether the action type was chosen in the demonstration dataset at state i.
4.4
RULE-BASED ACTION SELECTOR
4.4.1
ABSTRACT SUPPORTING EDGE SETS
When the input a knowledge graph, the action naturally strongly correlated with some critical edges. For example, ("potato", "needs", "diced") and ("potato", "in", "player") can lead to the action "dice potato". We refer those decisive edges correlating to an action as the supporting edge set of this action. Since have grouped actions by their semantic meanings, actions within each action type are actually supported by similar edges. For example, "dice potato" is supported by ("potato". "in", "player"), and "dice apple" is supported by ("apple", "in", "player"). As mentioned in Sec 4.c each action type comes with an action template like "dice object". Based on the action template, perform
sorts of abstraction. For example, given an input knowledge graph labeled with action "dice apple". we can replace all the "apple" appearing in the graph edges and the action with an abstract name "object". Then we can say the action "dice object" essentially supported by ("oojeet", "in",, "player"), where the two "object" should be instantiated by the same word. Under this kind of abstraction, different actions within the same action type can share a same abstract supporting edge set which contains edges with abstract names.
The abstract supporting edge set indicates the decisive edges for an action type, and it can be instantiated for each specific action. For example, check whether the action "dice apple" should be executed, the abstract edge ("object"" "in". "player") will be instantiated to edge ("apple", "in", "player"). Then, the existence ("apple", "in", "player") in input knowledge graph becomes an evidence for selecting the action "dice apple". We aim at finding an abstract supporting edge set for each action type, and will be used during inference.
4.4.2 MINE ABSTRACT SUPPORTING EDGE SETS FROM DEMONSTRATIONS
Finding the abstract supporting edge set for each action type is actually rule mining process. There are several off-the-shele rule miners like FP-Growth (Han et al. 2004). Apriori (Agrawal et (9944) Eclat (Zaki 2000), etc. but they not designed for knowledge graphs. Thus, propose simple yet effective rule miner for our setting to discover the supporting edge sets.
To find the Abstract Supporting Edge set ASE(AA) for each action type Ag, we designed numerical statistic that is intended to reflect the importance of an edge when taking an action a E A, and this numerical statistic is inspired by if-idf (Rajaraman & Ullman 2011). Formally, for action type Ag. we have a subset of demonstration dataset Dr { (ignoring тi here). Under the abstraction mentioned above, we can count the edge frequency for every (abstract) edge e in DR:
(ss]/s; E DR, E s:Hl
freq (e)
|Dxl
Similarly, we can also count the edge frequency for the entire demonstration dataset:
freg(e) Hs E D, stH
[D]
And we can define an importance score of an edge w.n.t to the action type Ag:
(+(e) frequ(e) log( Freg(e)
5

---

Under review as a conference paper at ICLR 2022
where the term frequ(e) is similar to the cern-freuuecy (tf), and the term log( freq(e) is similar to the inverse document frequency (idf).
Then we can get the ASE( As) by selecting the edges with the importance higher than a threshold, ie., ASB(AA) {e] (e) ) r}. where a hyperparameter shared across all action types.
4.4.3 INFERENCE BASED ON SUPPORTING EDGES
During inference, we use the supporting edge sets to score each action within the action type provided by action pruner, and select the action with the highest score. Figure 2 shows a concrete example of using supporting edge set to score an action.
Firstly, the action pruner outputs an action type based on the input knowledge graph, and we can retrieve the abstract supporting edge set of this action type. Secondly, given an action within the action type, we can instantiate the abstract supporting edge set specific supporting edge set by replacing the abstract names by the concrete words according to the action, e.g. ("object", "in", "player") will be instantiated to ("potato", "in", "player") the action is "cook potato with oven". Finally, we can compare the input knowledge graph with the supporting edge set of each action to figure out which supporting edge set is best covered by the input knowledge graph. The number of overlapped edges between the supporting edge set and the the input knowledge graph will be regarded as the score of the action.
Formally, the inference process of our rule-based action selector can be described follows
fs (s, A)
arg max |s n SE(a)), .Ел
where SE(a) is the supporting edge set associated with action which is obtained by instantiating the abstract supporting edge set of action type A.
Action Type
Abstract Supporting Edge Set
Action
cook
[object, needs, cooking method] [object, in, player]
cook potato with
Action Pruner
Input Knowledge Graph
Supporting Edge Set
Action Score
[player, at, kitchen] [potato, needs, roasted] [fridge, is, open] [potato, player]
[polato, needs, roasted] [potato, in, player]
Figure 2: An example of using supporting edge set to score an action. Based on the knowledge graph, the action pruner first predict the action type (e.g., cook) that needs to be taken current state, then instantiate the abstract supporting edge set to a concrete supporting edge set. Comparing the input knowledge graph with the supporting edge set, can compute action score for each action and select the action with highest score accordingly.
EXPERIMENTS
5.1 DATASET SETUP
We evaluate our method on TextWorld, which framework for designing text-based interactive games. More specifically, we use the TextWorld games generated by GATA (Adhikari et al. (020). In these games, the agent asked to cook meal according to given recipes. requires the agent to navigate among different rooms to locate and collect food ingredients specified in the recipe, process the food ingredients appropriately, and finally cook and eat a meal.

---

Under review as a conference paper at ICLR 2022
Difficulty Level
Recipe Size
Number of
Need
Need Cook
Number of Action Candidates
Number of Objects
Locations
Cut
1
Yes
No
11.5
17.1
1
Yes
Yes
11.8
17.5
1
No
No
7.2
34.1
28.4 Table TextWorld games statisties (averaged across all games within difficulty level). The games and statistics are generated by (Adhikari et al., 2020).
3
Yes
Yes
33.4
The state received by the agent is knowledge graph describing all the necessary information about the game. All the nodes and edges in the knowledge graph are represented in text. Figure shows partial example of input knowledge graph. The actions are also represented text. Note that the number of available actions vary from state state, so most of existing network architecture used deep reinforcement learning cannot be directly used here.
uncut
fried
diced
needs
needs
is
is
raw
Orange hell pepper
Yellow bell
part of
pepper
needs
The games have four different difficulty levels, and each difficulty level contains 20 training, 20 validation, and 20 test environments, which are sampled from a distribution based on the difficulty level. The higher the difficulty levels are, the more complicated recipe will be and the more rooms food ingredients will be distributed among. Statistics of the games are shown Table [ For evaluating model generalizability, we select the cop-perforning agent on validation sets and report its test scores; all validation and test games are unseen in the training set.
Fridge
fried
Cookbook
is
needs
part
closed
in
raw
uncut
is
onion
Figure 3: Visualization of part of an input knowledge graph in TextWorld.
5.2
IMPLEMENTATION DETAILS
5.2. PROCESS KNOWLEDGE GRAPHI BY GNNS
Graph neural networks (GNNs) serve as backbone networks in both our RL teacher policy and the action pruner our two-step hybrid policy. More specifically, we use Relational-Gi -CCN to take edge attributes into consideration.
The relations and nodes of the input knowledge graphs are initially represented text (words or short sentences). We use fastText (Mikolov et al. (007)), pre-trained, on Common Crawl, to convert each token an embedding. and then average all the token embeddings to obtain the mean fastText embedding of the entire text string. These mean fastText embeddings are pre-computed and fixed during training. For each relation and node, we append their corresponding mean fastText embedding with trainable embedding vector. The concatenated embeddings then used as the numerical representation of each relation and node, which are further passed through GNNs to get the latent embedding of the entire input knowledge graph.
5.2.2 DEMONSTRATION ACQUISITION
To collect demonstration dataset, first train teacher policy by DQN (Mnih et al. (0155) in the training environments, which can converge to a near-optimal solution. adopt variable number of available actions, we let the policy network take the state and available action input and output a score for this action. And we can select the action with maximum score as the final output.
The trained teacher policy is used collect 300K samples through the interaction with the envi- ronment, and label them with the taken actions, as illustrated in Sec 4.2.1 When collecting the demonstration dataset, use e-greedy exploration strategy increase the diversity of states.
'The code reproduce all results in this work will be publicly availahle.
7

---

Under review as a conference paper at ICLR 2022
Action Type: Cut
Example Action
Action
verb object with knife
slice potato with knife
Head
Relation
Tail
Importance
Head
Relation
Tail
in needs part of object is
object
player
1.7004
potato
in needs
player
Discovered
object
verb serb_passive
1.6659
potato
sliced
Supporting
Edges
object
cookbook 1.0058
potato
part of cookbook
uncut 0.9837
potato
uncut
Table 2: Discovered abstract supporting edges of the "cut" action type in difficulty level of TextWorld environments. The importance of each edge is computed by the metric mentioned in See 4.4.2 The right column gives an example action and the corresponding supporting edge set, which is obtained by instantiating the abstract supporting edge set. The underlined words are abstract names for some certain words. For example, "verb" in action can be instantiated to "slice", "dice" or "chop", and "verb "verb_passiii passive" can be instantiated to "sliced", "diced" or "chopped" accordingly.
5.3 RESULTS
5.3.1 INTERPRETABILITY
The interpretablity of our two-step policy is two-fold: 1) the transparent two-step decision making process; 2) the rule-based models make decisions in a way which is easy to interpret by human.
Table 2 shows some representative rules discovered by our rule miners. We observed that all of the four discovered abstract supporting edges are indeed prerequisites of the "cut" actions. In particular, the abstract supporting edge ("object", "needs". "verb verb-passivve") is crucial prerequisite for the agent to select the correct verb the "cut" actions. For example, if the input knowledge graph contains the edge ("potato", "needs", "sliced"), then the action "slice potato with knife" will get one than others because this edge is in the supporting edge set of action "slice potato with knife".
score
Since we clearly see the rule-based model makes decisions based these rules, it is not hard to check whether this model works in a correct way. In this way, the agent can certainly select the correct "cut" action even in unseen test environments.
5.3.2 GENERALIZATION
Training
Test
Difficulty
3
4
GATA-GTF
98.6
58.4
95.6
36.1
83.8
53
33.3
23.6
Vanilla RL
100
100
98.3
100
83.8
68
50
30.9
Ours
100
100
100
65.5
100
100
51.7
49.7
Table Evaluation results on both training environments and test environment in TextWorld. The numbers show the agent's normalized scores.
We compare with two baselines: GATA-GTH and vanilla RL. GATA-GTF is variant from GATA (Adhikari et al. 2020). and uses the same ground-truth graph input as us. GATA-GTF processes the input graphs by relational graph convolutional networks (R-GCNs) (Schlichtkrull et al. 2018), and the whole pipeline trained by DQN (Mnih et al.) 2015). Vanilla RI is essentially the same method with GATA-GTE. but implemented by ourselves. With better implementation and careful hyperparameter tuning, it performs better than the implementation released by the GATA authors. And vanilla RL also serves the teacher policy which used to generate demonstrations for our method.
Table 3 shows the normalized scores of different methods on both training environments and test environment in TextWorld. The result of GATA-GTF is obtained from its paper (Adhikari al. 2020 In all the enviromments, our agent achieves better generalization performance the vanilla RL (which our RL teacher) and GATA-GTF baselines. Our agent can generalize pretty well to the
8

---

Under review as a conference paper at ICLR 2022
Noise
Difficulty
Add
Drop
RL
Ours
RL
Ours
RL
Ours
RL
Ours
0.2
0
100(0%)
10000% %) 100(0%)
100(0%)
96(-1 %6)
98(-1 %)
38(-61%)
64(-1%) )
0.2
0.03
100(0%)
96(-3%)
81(-19%)
98(-1 %)
80(-18%)
98(-1 %)
41(-58%)
51(-21%)
0.2
0.06
100(0%)
92(-7%)
93(-6%)
86(-11%) 82(-18%)
93(-6%)
44(-55%)
41(-36%)
0.4
100(0%)
100(0%)
85(-15%)
100(0%)
65(-33%)
91(-8%)
19(-80%)
58(-11%)
0.4 0.03
100(0%)
100(0%)
77(-23%)
90(-9%)
71(-27%)
91(-8%)
20(-79%)
53(-18%)
0,4 0.06
100(0%)
96(-3%)
78(-22%)
83(-16%)
65(-33%)
96(-3%)
19(-80%)
49(-24%)
0.6 0
100(0%)
100(0%)
85(-15%)
100(0%)
76(-22%)
93(-6%)
8(-91%)
59(-9%)
0.6 0.03
100(0%)
97(-2%)
83(-16%) 75(-25%)
61(-37%)
86(-13%)
9(-90%)
39(-39%)
0.6
0.06
100(0%)
91(-8%)
76( (66)244)
80(-20%)
60(-38%)
96(-3%)
10(-89%)
46(-29%)
Table 4: Robustness analysis of our method and vanilla RL baseline. We evaluate the performance of agents on training environments under different noise levels, e.g., "add 0.2 drop 0.03" means we randomly add 20% additional edges while randomly dropping 3% existing edges in graph. The numbers out of parentheses are normalized scores and the numbers in parentheses are relative performance change comparing the performance without input noise. The bold numbers indicate which method performs better in each setting.
unseen test environments in all difficulty levels, while the performance of GATA-GTF and vanilla RL performs poorly in unseen test environments.
5.3.3 ROBUSTNESS
As mentioned above, our method also aims at robustness. To evaluate the robustness of our models, we add different levels of noises to input knowledge graphs and regard the performance of the agents under noisy inputs. In this paper, we define the noise on knowledge graph as adding additional edges to the graph or dropping existing edges in the graph. Formally, we add noise (k:, plevel to input knowledge graph in the following way:
1. Add edges: [K [T] additional edges will be randomly generated and added to the graph, where E is the edge set of the graph. For each edge h;, 4, r,, head node h2 and tail node 4 are sampled from Vau and relation r; is sampled from Rau, where Vau means the set of all possible nodes and Rattt is the set of all possible relations.
2. Drop edges: [p * [E]] edges will be dropped from the graph. Note that only original edges will be dropped, and the randomly added edges will not be dropped.
Table 4 shows the performance of vanilla RL and method under noisy input graphs generated in the above mentioned way. Note that we test robustness in training environments instead of test environments, because the performance of both agents in test environments are not good enough and examining robustness of poor-performed agents does not make too much sense.
We observed that in difficulty level 1, both agents very robust under the noisy inputs. However, in difficulty level 2 & & 4, the performance of the RL agents are hurts a lot by the input noises, especially in the difficulty level 4. In contrast, the agents obtained by our method are still performs pretty good and the performance drops significantly less than the RL agents.
6 CONCLUSION
In this work, we propose a two-step hybrid policy for graph-based reinforcement learning. The two-step hybrid policy disentangles complicated black-box deep reinforeement learning policy into an action pruner and an action selector. The joint effort of two modules can not only generate human- -finndlf explanations of each decision, but also provide significantly better performance in terms of generalization and robustness. We conduct comprehensive experiments on the representative text-based environments (Côté et al. 2018) to demonstrate the effectiveness of our approach.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ashutosh Adhikari, Xingdi Yuan, Marc-Alexande Côté, MikuláK Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and Will Hamilton. Learning dynamic belief graphs to generalize on text-based games. Advances in Neural Information Processing Systems, 33, 2020.
Rakesh Agrawal, Ramakrishnan Srikant, et al. Fast algorithms for mining association rules. In Proc. 20th int. conf. very large data bases, VLDB, volume 1215, pp. 487-499. Citeseer, 1994.
Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp- 2499-25999 2018.
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological slam for visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 12875-12884, 2020.
Chih-Chou Chiu, Ling-Jing Kao, and Deborah F Cook. Combining a neural network with rule-based expert system approach for short-term power load forecasting in taiwan. Expert Systems with Applications, 13(4):299-305. 1997.
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Workshop on Computer Games, pp. 41-75. Springer, 2015 8.
Mengnan Du, Ninghao Liu, and Xia Hu, Techniques for interpretable machine learning. Communica- lions of the ACM, (36-7777777 2019.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference on Learning Representattonss, 2018.
Rodney M Goodman, Chuck Higgins, John Miller, and Padhraic Smyth. A rule-based approach to neural network classifiers. In International Neural Network Conference, pp. 886-889. Springer, 1990.
Hayit K. Greenspan, Rodney Goodman, and Rama Chellappa. Combined neural network and rule- based framework for probabilistic pattern recognition and discovery. 1992.
Jiawei Han, Jian Pei, Yiwen Yin, and Runying Mao. Mining frequent patterns without candidate generation: A frequent-pattern tree approach. Data mining and knowledge discovery, 8(1):53-87, 2004.
Marcel Hildebrandt, Hang Li, Rajat Koner, Volker Tresp, and Stephan Günnemann. Scene graph reasoning for visual question answering. arXiv preprint arXiv:2007.0100 2020.
De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, and Juan Carlos Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demonstration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8565 8574, 2019.
Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob Pettit, and Daniel Faissol. Discovering symbolic policies with deep rein- forcement learning. In International Conference Machine Learning, pp. 5979-5989, PMLR, 2021.
Andrew Levy, George Konidaris, Robert Platt. and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference on Learning Representationss 2018.
Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. Toward interpretable deep reinforcement learning with linear model u-trees. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 414 429. Springer, 2018.
10

---

Under review as a conference paper at ICLR 2022
Daoming Lyu, Fangkai Yang. Bo Liu, and Steven Gustafson. Sdrl: interpretable and deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2970-297777 2019.
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2493-2500, 2020.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. Advances in pre-training distributed word representations. arXiv preprint (xXiv:1122.04005 2017.
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. A graph placement methodology for fast chip design. Nature, 594(7862)2207-21,, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529 -23-533,, 2015.
Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. In Advances in Neural Information Processing Systems, pp. 12329-12338, 2019.
Tongzhou Mu, Jiayuan Gu, Zhiwei Jia, Hao Tang, and Hao Su. Refactoring policy for compositional generalizability using self supervisd object proposals. Advances Neural Information Processing Systems, 33, 2020.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 31:3303-3313, 2018.
Yuzuru Okajima and Kunihiko Sadamasa. Deep neural networks constrained by decision rules. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2496-2505, 2019.
Lauv Patel, Tripti Shukla, Xiuzhen Huang, David W Ussery, and Shanzhi Wang. Machine learning methods in drug discovery. Molecules, 25(22):5277, 2020.
Erika Puiutta and Eric MSP Veith. Explainable reinforcement learning: A survey. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction, pp. 77-95. Springer, 2020.
Anand Rajaraman and Jeffrey David Ullman. Mining of massive datasets: Data mining (ch01). Min. Massive Datasets, 18:1 14-142, 2011.
Paramita Ray and Amlan Chakrabarti. A mixed approach of deep learning method and rule-based method improve aspect level sentiment analysis. Applied Computing and Informatics, 2020.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 155):206-215, 2019.
Michael Schlichtkrull, Thomas Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pp. 593-607. Springer, 2018.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward I Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):044669 2020.
Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. arXiv preprint arXiv: 2017.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7787):484 489, 2016.
11

---

Under review as a conference paper at ICLR 2022
Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim. Neural program synthesis from diverse demonstration vidcos. International Conference on Machine Learning, pp. 4790- 4799. PMLR, 2018
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pp. 5045-5054. PMLR, 2018.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu, Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. 3540-3549, PMLR, 2017.
Tong Wang. Gaining free or low-cost interpretability with interpretable partial substitute. In International Conference Machine Learning, pp. 6505-6514. PMLR, 2019.
Vikram Waradpande, Daniel Kudenko, and Megha Khosla. Graph-based state representation for deep reinforcement learning. arXiv preprint arXiv: 13965, 2020.
Mohammed Javeed Zaki. Scalable algorithms for association mining. IEEE transactions on knowledge and data engineering, 12(3)372--300 2000.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap. Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. 2018.
12


Reviews:
[{"review_id": "y4mfk1blvXp", "reviewer": "Reviewer_3CUH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, I am worried that this framework is not very generalizable outside of this specific problem.\n\nFirst, as the authors acknowledge, it only makes sense for graph-based inputs where there is no real perception. The input just tells you what objects are available. This method would not work at all in any other kind of environment, such as one with visual perception instead of symbolic input, in continous/physics based environments, or even games such as atari where there aren't symbolic inputs and graphs. This is already a pretty narrow problem.\n\nOn top of that, this method is only tested on one extremely specific subset of all graph symbolic-input environments relating to cooking on a single dataset. And cooking seems to be an extremely limited domain (mostly a fixed set of the kinds of actions you can do with a relatively small set of objects in a small space). On average, how many steps are required for each of these difficulties? With the graph input nature of the dataset and how many steps are required in the recipe, it seems like the GT policy would be really really short, right? Does this method work on other TextWorld games, perhaps ones with more different kinds of actions? Or environments that also have graph inputs such as VirtualHome?\n\nThere is also a detail that is really briefly mentioned in the paper that is concerning. Specifically what is the method is selecting this set of K different action types and how the ground truth label k_i is selected for each base action? The paper says that \"the action type k_i is determined by k_I = h(\\pi(s_i)) where h is a domain specific function which takes the semantic meaning of the action into consideration.\" This is not at all clear, but it sounds like it's saying that this is a hand-crafted rule by the designed. In which case, I would argue that is putting way too much of the specifics of the environment as hand-crafted features. This means that if we have another environment, for instance a graph-based battle simulator in textworld that a human designer would need to go in and hand-craft parsing rules for all of the possible actions the agent could do in that environment and so on and so on for any new task you want to add. This seems very unscalable and putting a heavy thumb on the scale.\n\nThis action grouping idea also seems quite specific to text adventure games or environments where the action spaces is language. This grouping actions by their specific type does not seem to really occur in any other environment I can think of other than text-based or purely symbolic environments. \n\nThere are by this point very good methods that show generalization results on all of TextWorld and on other text adventure games. With train/test splits occurring across completely different categories of games, and without the restriction of only graph inputs. There are much more general methods which perform extremely well in these environments, so I am not entirely sure how useful a line of work is on this very narrow subproblem.\n\nFor evaluation, there should also be more comparison to state-of-the-art methods in TextWorld. GATA-GTF is a good baselines because it also was developed on this specific subset of TextWorld, and the RL baseline is also good. But there are many other methods on TextWorld that also do not have this constraint on graph inputs and are demonstrated on all of TextWorld. There should be at least one or two comparisons to one of these SOA methods.\n\nMinor: Is the Vanilla RL actually the same as the policy used by the method to obtain the teacher policy? If not, could you show the results of the teacher policy in Table 3, that seems like an interesting comparison to make and it's already available.\n\nOn the positive side, I did find the edge set mining part of the method interesting, although I'm not sure whether it is a method that is very applicable in non-graph-based inputs.\n", "summary_of_the_paper": "This paper introduces a two-step method for RL problems with graph-based inputs. This method trains a teacher policy which it then uses to construct a policy \"pruner\" which predicts the kind of action needed. It then uses a ideas from retrieval to select the edges needed to complete the action.", "summary_of_the_review": "The method is interesting, but it seems very contrived for a very small and specific type of problem. There appear to be a lot of components which would not generalize even to very similar TextWorld domains. The result is only tested on one small subset of TextWorld and the state-of-the-art comparisons are not adequate.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qtUYpANMOhK", "reviewer": "Reviewer_5Hfh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## What I like about this paper\n\n- I like to see research on RL applied to graph data. I believe several embodied environments can be abstracted and represented as graphs on which to learn generalizable and interpretable policies. TextWorld environments seem a good testbed for that.\n\n- Reducing the search space by breaking down the potential actions in groups sharing similar semantics is a neat idea. I believe this could be learned or even extracted from large language models like GTP-3.\n\n\n## Concerns\n\n- Looking at the literature around text-based games, I found relevant references that seem to be missing: Action-Elimination [1], T-DQN [2]. Since the proposed approach is trying to reduce the action space, I believe those would be decent baselines to compare against.\n\n- It is not clear to me if the proposed approach is performing well because of the amount of prior knowledge (or expert domain knowledge as said in the paper) that has been injected into the model. Regarding the action selector, instead of using a rule-based approach, maybe some attention mechanism over the edges and nodes of the graph could be used?\n\n- In Table 2, shouldn't there also be supporting edges that contain the \"knife\", e.g. the knife should be held by the player?\n\n- In Table 3, why is the Vanilla RL approach getting 100 on training level 4, which is much higher compared to the two other models?\n\n- Section. 5.3.3. Robustness. What happens if the edges related to the goal are dropped (e.g, the 'needs' edges from Figure 3)?\n\n#### References\n- [1] Zahavy, Tom et al. “Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning.” NeurIPS (2018).\n- [2] Hausknecht, Matthew J. et al. “Interactive Fiction Games: A Colossal Adventure.” ArXiv abs/1909.05398 (2020)\n\n### Minor\n\n - p.3: \"In text-based games, the agent receives a knowledge graph...\" -> By their nature, text-based games' input modality is text. Instead, I would mention TextWorld provides the underlying state of the game represented as a graph, or something along those lines.\n\n - p.4: \"2) Furthermore, this step also eliminates the search space to enable a feasible candidates set for the rule-miner.\" I don't fully understand what that means.\n\n-----\n### Typos\n- p.3: The sentence \"Therefore various ... models.\" is missing a verb.\n- p.3: \"... usually more than an order of magnitude smaller than original actions\". I find it hard to parse \"more than ... smaller\". Just stating $K << |A|$ seems enough to me.\n- p.3: \"... action type $A_i$.\" -> Since $i$ refers to the time step, I don't think that makes sense to use it. Earlier, it was stated that $k = f_p(s_i)$, so $A_k$ might be more appropriate, or $A_{k_i}$ as used later in the text.\n- p.4: \"end-to-end differential\" -> differentiable\n- p.4: \"we target at\" -> \"we aim at\"?\n- p.4: \"After obtaned\" -> \"After obtaining\"\n- p.4: \"in demonstration dataset\" -> \"in the demonstration dataset\"\n- p.4: \"Therefore, we needs\" -> need", "summary_of_the_paper": "This paper is about improving the interpretability of agents trained on reinforcement learning problems where the input observation is given as a graph. Specifically, the authors propose to split the decision-making process in two: 1) an action pruner, and 2) an action selector. Experiments were conducted on a suite of TextWorld environments that have their state exposed as a knowledge graph. Empirically, it was shown that the proposed two-step approach is better at generalizing on unseen environments and is more robust to noisy perturbations on the input graphs.", "summary_of_the_review": "Dealing with large action is an important research direction in RL. I find the proposed approach interesting, intuitive, and allows for better interpretability. While the results are encouraging, I'm concerned with the amount of prior knowledge that went into this, e.g. semantic grouping of the actions, and rule-based action selector. Could those priors be learned? Also, I believe this paper could be strengthened by evaluating their approach on other types of tasks (i.e., not related to cooking) from the TextWorld suite. For those reasons, I can't recommend it for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XlU9V7dP9rc", "reviewer": "Reviewer_eXdP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points of the paper: (i) use of more powerful representations (graphs) to describe the task and states; (ii) generating decisions that can be explained in a human-friendly representation; (iii) dividing the problem into two levels (first mapping the states into abstract actions and then mapping these actions into primitive actions); (iv) interesting proposal to mine the ASEs (GNN-Abstract Supporting Edge sets) from the demonstrations given by the teacher.\n(v) good results in complex text-based games; (vi) clear and well-written paper.\n\nNegative points of the paper: (i) why does the paper refer to interpretable RL (even in the title), since the proposal is basically a combination of supervised (SL) and unsupervised (UL) learning? (ii) The related works do not, in fact, provide an overview of decision-making techniques that combine SL and UL. (iii) It wasn't clear to me how h(.) is defined (function that groups actions) - the definition of this function seems to be quite manual and domain dependent, which restricts its use of the proposal. (iv) It wasn't clear to me exactly what the inputs are: In the pipeline in Fig1 it indicates that the state (given in knowledge graph - KG) is the input, but in the text, it indicates that KG also describes the task. Clarify this further in the text.\n", "summary_of_the_paper": "This paper proposes a pipeline to determine (primitive) actions to be performed in states described by knowledge graphs. The pipeline is divided into two steps. The first step, Action Pruner, maps states into abstract actions (i.e., clusters of primitive actions) and is performed by a trained GNN with data from a teacher. The second step, Action Selector, is a rule-based system that defines the primitive action to be performed among those included in the abstract action given by the previous step. The Action Selector defines the primitive action based on the GNN Abstract Supporting Edge set (ASE), also defined from the training data provided by the teacher. The proposal presented good results in games with text-based environments and also allows the generated decisions to be human-friendly explanations.", "summary_of_the_review": "The paper reports a method that decides an action (textual) based on a state (and task) described by a knowledge graph. The idea is interesting, and the results are significant, in addition to enabling the interpretation of decisions made by the system. However, the scope of the proposal seems limited and, despite being well written, some points of the proposal should be further clarified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BlfI5o6CCEI", "reviewer": "Reviewer_WZ15", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The idea is novel. The proposed method provides a kind of post-hoc explanation where graph-based model with interpretability is learned based on the demonstrations generated by a decent good policy of any DRL (without consider its interpretability).\n- The topic is more like a timely important topic.\n- The paper is well-written.\n\nWeakness:\n- Graph-based model can provide interpretability based on causal relations, but the authors didn't talk much about this perspective regarding their own model. Instead, the authors demonstrate more interpretations to the predictive outcomes. This undermines the interpretability of the proposed method.\n- Action group: according to the example of actions, it looks the current model is more suitable to handle the task like text games. I am concerned how it works if the proposed method applies to other domains, like robot control or atari games. Can you elaborate it with an example? If the proposed method currently only can handle the text games, it applicability on other domains is limited.\n- regarding the empirical evaluation for generalization, robustness, it looks the baselines are selected improperly and the comparison is unfair. For example, the performance of GATA-GTF in Table 3 is bad and far from Vanilla RL, why select it as a baseline? Also, it is not suprising that Vanilla RL (as said by) doesn't have generalization ability. I also have the same questions for the comparison in Table 4.\n\n\nMinors:\n- \"hiereachical\" in sec.2.2 should be \"hierarchical\".\n- \"on one hand\" in sec.3 -> \"on the one hand\"\n- For the action type $A_k = \\{ a_k^1, \\ldots, a_k^n \\}$ in sec.3, what does this \"n\" mean?\n", "summary_of_the_paper": "This work targets to the problem of limited interpretability in RL. To solve the problem, the authors propose a two-step hybrid model, including action pruner and action selector. The proposed method is evaluated on the text games, with results showing its ability of interpretation, generalization, and robustness.", "summary_of_the_review": "-The contribution of this work is primarily the empirical one. \n-The authors aim at achieving the interpretability, generalization ability, and robustness of their methods. However, due to the concerns I raised in the weakness part, I prefer to weak reject (marginally below the acceptance threshold) the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors did not respond to the concerns raised by all the reviewers. As the recommendation were on the edge, this lack of engagement seems odd, and it left the reviewers with little material to discuss and revise their recommendation. We recommend the authors carefully consider the reviews if they plan to resubmit.
Decision: Reject