Inherently Explainable Reinforcement Learning in Natural Language
Authors: XIANGYU PENG
Mark Riedl
Prithviraj Ammanabrolu
Venue: NeurIPS 2022 Accept (2022)
Keywords: knowledge graph
reinforcement learning
explainable AI
natural language processing

Abstract: We focus on the task of creating a reinforcement learning agent that is inherently explainable---with the ability to produce immediate local explanations by thinking out loud while performing a task and analyzing entire trajectories post-hoc to produce temporally extended explanations. This Hierarchically Explainable Reinforcement Learning agent (HEX-RL), operates in Interactive Fictions, text-based game environments in which an agent perceives and acts upon the world using textual natural language. These games are usually structured as puzzles or quests with long-term dependencies in which an agent must complete a sequence of actions to succeed---providing ideal environments in which to test an agent's ability to explain its actions. Our agent is designed to treat explainability as a first-class citizen, using an extracted symbolic knowledge graph-based state representation coupled with a Hierarchical Graph Attention mechanism that points to the facts in the internal graph representation that most influenced the choice of actions. Experiments show that this agent provides significantly improved explanations over strong baselines, as rated by human participants generally unfamiliar with the environment, while also matching state-of-the-art task performance.

Inherently Explainable Reinforcement Learning in Natural Language
Xiangyu Peng
Chen Xing
Prafulla Kumar Choubey' Chien-Sheng Wu, Caiming Xiong
'Georgia Institute of Technology (xpeng62, ried1 3@gatech. edu, raja@allenai. org
+Allen Institute for
Abstract
We focus on the task of creating reinforcement learning agent that inherently explainable-witt the ability to produce immediate local explanations by think- ing out loud while performing task and analyzing entire trajectories post-hoc produce temporally extended explanations. This Hierarchically Explainable Reinforcement Learning agent (HEX-RIJ) operates Interactive Fictions, text- based game environments in which agent perceives and acts upon the world using textual natural language. These games are usually structured as puzzles quests with long-terre term dependencies in which agent must complete a quence of actions to succeed proocceedpoovdding providing ideal environments in which to test an agent's ability to explain its actions. Our agent is designed to treat explainability as a class citizen, using an extracted symbolic knowledge graph graph-based (KG) state representation coupled with Hierarchical Graph Attention mechanism that points to the facts in the internal graph representation that most influenced the choice of actions. Experiments show that this agent provides significantly improved explanations over strong baselines, as rated by human participants generally unfamiliar with the environment, while also matching state-of-the of-tt--r-tt task performance.
Introduction
Observation: Forest Path
The path heads northsouth here.
One particularly large tree with some low branches stands the edge of the path...
Explainable Al refers to artificial intelligence methods and techniques that provide human- understandable insights into how and why AI system chooses actions or makes predictions. Such explanations critical for ensuring reliabil- ity and improving trustworthiness by increasing user understanding of the underlying model. In this work we specifically focus on creating deep re- inforcement learning (RI.) agents that can explain their actions in sequential decision making envi- ronments through natural language.
Action: go up
Game Score:
Explanation:
O
in the
Path
frees are intereaclable
o Agent needs this action to:
egg at Up
Observation: Up a tree Beside you the the birds nest is large egg encrusted with precious jewela, scavenged by a childless songbird...
small birds
Action: take egg
Game Score: 5
Explanation:
egg interactable
In contrast to the majority of contemporary work in the area which focuses supervised machine learning problems which require singular instance
Observation: Up
Taken.
level local explanations You al. 2016 Xu al.
Figure 1: Excerpt from zorkl with immediate step-by-step explanations constructed from the KG represented by and temporally ex- tended explanations by 2 Colors represent different categories of KG facts seen Fig
2015 Wang et al. 2017 Wiegreife and Marasovic
2021 such environments in which agents need
to reason causally about actions over a long series
Code:
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

of steps require an agent to take into account both environmentally grounded context as well as goals when producing explanations. Agents implicitly contain beliefs regarding the down- stream effects- changes to the world-ther actions taken at the current timestep will have. This requires explanations in these environments to contain an additional temporally extended component taking the full trajectory's context into account compunttcmm to the immediate step-by-step explanations.
Interactive Fiction (IF) games (Fig, are partially observable environments where an agent perceives and acts upon a world using potentially incomplete textual natural language descriptions. They are structured as long puzzles and quests that require agents to reason about thousands of locations, characters, and objects over hundreds of steps, creating chains of dependencies that an agent must fulfill to complete the overall task. They provide ideal experimental test-beds for creating agents that can both reason in text and explain it.
We introduce an approach to game playing Ceeents-HHeerrhhhccall Explainable Reinforcement Learning (HEX-RLL RL)- is designed be inherently explainable, in the sense that its internal state representation-i.e. belief state about the world-takes the form of a symbolic, human- interpretable knowledge graph (KG) that is built the agent explores the world. The graph is encoded by a Graph Attention network (GAT) (Veličković et al. 2017 extended to contain a hierarchical graph attention mechanism that focuses on different sub-gra graphsrphs in the overall KG representation. Each of these sub- -ub-ggaahs contains different information such as attributes of objects, objects the player has, objects in the room, current location, etc. Using these encoding networks in conjunction with the underlying world KG, the agent is able to create immediate explanations akin to a running commentary that points to the facts within this knowledge graph that most influence its current choice of actions when attempting to achieve the tasks in the game on a by--tpp basis.
While graph attention can tell us which elements in the KG are attended to when maximizing expected reward from the current state, cannot explain the intermediate, unrewarded depen- dencies that need to be satisfied to meet the long term task goals. For example, in the game zork1, the agent needs to pick up lamp early on in the game an unrewarded action- but the lamp is only used much later on to progress through a location without light. Thus, our agent additionally analyzes an overall episode trajectory- a sequence of knowledge graph states and actions from when the agent first starts in world to either task completion or agent death- to find the intermediate set of states that are most important for completing the overall task. This information is used to generate temporally extended explanation that condenses the immediate aep-by-step explanations to only the most important steps required to fulfill dependencies for the task.
Our contributions are as twofold: (1) we create an inherently explainable agent that uses an ever-updating knowledge-grae -raph based state representation to generate step-by-step immediate explanations for executed actions as well as performing post-hoc analysis to create temporal explanations; and (2) a thorough experimental study against strong baselines that shows that our agent generates significantly improved explanations for its actions when rated by human participants unfamiliar with the domain while not losing any task performance compared to the current state- of- the-ar aht knowledge graph- based agents.
Background and Related Work
Interactive Fiction (IF) games are simulations featuring language- based state and action spaces. provides a platform for exploring lifelong open-domain dialogue learning (Shuster et al. (220) and action elimination with deep reinforcement learning (Zahavy et al. 2018). In this paper, we use IF games as our test-bed because they provide an ideal platform for collecting data, linking game states and actions to the corresponding natural language explanations. We use the definition of text-adyentur games as seen in Côté et al. (2018) and Hausknecht et al. 2020). We take Jericho (Hausknecht et al. 2020), a framework for interacting with text games, as the interface connecting learning agents with interactive fiction games. A text game can be defined as a partially-observan Markov Decision Process: G = (S, P, 0, R, Y), representing the set of environment states, conditional transition probabilities between states, the vocabulary or words used to compose text
2

---

f-Action
Velue
Vt
Siquental Action
Value
Predictor
Actor
Dacadar
udue
Offferacchiiaal
Herrachical
LSTM Attenttin
Attention
Fizuee
LLSTM
gr
g
vybii e dEgeyyeooo ong
hoow
LSTM
GAT
GAT
Hierarchical
cgp
Key: Otis
Updane
G
Updato
Dileas the
Inu Ciga Affituttes
House
Recurent Text Encoder
G
Knowledge Graph
Behind House
house. path ladd the forest to the cast. one corner of the house (heve small window which slightly ajar. carrying: jewel-encrusted egg Attributes: ta talkable, openable, animate, treasure...
behind
Stall
Ot
Queatooa::
Regresomiation
ALBERT-QA
(Wiatt dos Iiaus...
Observation
Figure 2: Knowledge graph extraction and the (HEX-RL) agent's architecture at step 1.
commands, observations, observation conditional probabilities, reward function, and discount factor, respectively. The reinforcement learning agent trained to learned a policy (c( a.
Knowledge Graphs for Text Games. Ammanabrolu et al. 2020 proposed QEBERT, reinforce-
ment learning agent that learns a KG the world by answering questions. Xu at. 2020 introduce
the SHA-KG KG, stacked Hierarchical Graph Attention mechanism to construct explicit represen- tation of the reasoning process by exploiting the structure of the KG. Adhikari et al. 2020) present the Graph-Aided Transformer Agent (GATA) which learns construct KG during game play and improves zero-shot generalization on procedurally generated TextWorld games. Other works such as Murugesan al. (020)) explore how use KGs endow agents with commonsense, While these works showcase the effectiveness of KGs on task performance and do not focus on how explainable their architectures are. We further note that these architectures do now allow for as fine-grained attention based attribution as HEX-RL architecture does-e.g, Q'BERT does not use relationship information their policy and SHA- KG averages attention across large portions of the graph and is unable to point to specific triples in its KG representation to explain an action.
Explainable Deep RL. Contemporary work on explaining deep reinforcement learning policies can be broadly categorized based on: (1) how the information is extracted, either via intrinsic
motivation during training (Shu et al. 2017 Hein et al. 2017 Verma et al. 2018 or through
post-hoc analysis (Rusu et al. 2015 Hayes and Shah 201 7 Juozapaitis et al. 2019 Madumal
et al. 2020]; and (2) the scope either global Zahavy et al. 2016
Hein et al. |001 Verma et al.
2018: Liu et al., 2018) or local (Shu et al.) 2017 Liu et al. 20 18 Madumal et al. 2020 Guo et al.
2021). In our work, we create an agent that spans more than one these categories providing immediately local explanations through extracted knowledge graph representations and post-hoc temporal explanations. Inspired by Madumal et al. (2020), we learn a graphical causal model which focuses using relations between steps in puzzle to generate temporal explanations instead of generating counterfactuals.
3 Hierarchically Explainable RL
Our work aims to generate (1) immediate step-by-step explanations of agent's policy by captur- ing the importance of the current game state observation and (2) temporally extended explanations that take into context an entire trajectory via a post-hoc analysis. Formally, let = ar) be the set of game steps that compose a trajectory. Each game state s, consists of a knowledge graph G, representing all the information learned since the start of the game. This graph is further split into four sub- nub-knowledge graphs each containing different, semantically related relationship types. This section first describes a graph attention based architecture that uses these sub- sub-graphs to pro- duce immediate explanations. We then describe how to filter the game states in a trajectory into condensed set of the most important ones X' c that best capture the underlying dependencies that need to be fulfilled to complete the wask-enabling us to produce temporal explanations.
3

---

Knowledge Graph State Representation. Building on Ammanabrolu et al. 2020 constructing the knowledge graph is treated as a question-answeree task. KGs these games take the form of RDF triples (Klyne 2004) of (subject, relation, object)-extrateee extracted from text observations and update as the agent explores the world. The agent answers questions about the environment such as, "What am carrying?" or "What objects are around me?". A specially constructed dataset for question answering in text Jerines--ericooAA--- used to fine-tune ALBERT (Lan et al. 2019) to answer these questions (See Appendix A.3 The answers form set of candidate graph vertices Vi for the current step and questions form the set of relations R,. Both V, and R, are then combined with the graph at the previous step G,-1 to update the agent's belief about the world state into G₁- The left side of Figure showcases this.
In an attempt to enable more fine grained explanation generation and inspired by et al. 2020
we divide the knowledge graph G into multiple sub-graphs Ca , ,o ,e each represent- ing (1) attributes of objects, (2) objects the player has, (3) objects in the room, and (4) other information such as location (right side of Fig. based on the corresponding relationship types extracted by the ALBERT- -AA module. The union of all sub-graphs is equivalent of V, and R extracted from the current game state. The full knowledge graph G, captures the overall game state since the start of the game. The sub- uub-graphs easily reflect different relationships of the current game state.
Template Action Space. Agents output a language string into the game to describe the ac- tions that they want to perform. To ensure tractability, this action space can be simplified down into templates. Templates consist of interchangeable verbs phrases (VP), optionally fol- lowed by prepositional phrases (VP PP), e.g. (Icarry/ take] _ and ([throw/discarr engainst/ on/ down] J, where the verbs and prepositions within I.] are aliases. Actions are con- structed from templates by filling in the template's blanks using words the game's vocabulary. Size of action space is shown in Appendix A.
put]
3.1 Immediate Explanations
Our immediate explanations consist of finding the subset of triplets in sub-graphe Gatr, Gatr, Gine, gobj,C' c the action decision made at the current step-is capable of explaining the action. We introduce deep RL architecture capable of this.
Hierarchical Knowledge Graph Attention Architecture. At each step, total score R, and an observation 0, is sereived-consistin of (faesee Igame Ome a, corresponding to the room de- scription, game feedback, inventory, and previous action and are processed using a GRU based encoder using the hidden state from the previous step, combining them into a single observation embedding o, BBd (bottom of Fig. 2).
The full knowledge graph G, is processed via Graph Attention Networks (GATs) (Veičković et al. 2017) followed by a linear layer to get the graph representation g: e Rddest (middle of Fig. compute LSTM attention between o, and g as:
We
41STM = softmax (WIkISTM b))
(1)
hISTM = tanh W (Wggi bg))
(2)
where denotes the addition of a matrix and a vector. Wi E R Wg E R W, dheart are weights and by Rdbext, b, gheer are biases. The overall representation vector is updated as:
q gi E a1sTM, @ 0,,
where denotes dot- product and c is the number of o,'s components.

---

Sub-graph sub-graphs are also encoded by GATs to get the graph representation g [ (no. of
subgraphs). The Hierarchical Graph Attention between 9, E aR and & is calculated by:
Hierachicall = softmax (W bz)
(4)
hy = tanh (W' g' @ (W W.q."" bq))
(5)
where WH E R aqqaa Wg g' E su Wa R are weights and by Ru by Ra are biases. Then we get state representation, consisting of the textual obser- vations full knowledge graph and sub-knowledge graph.
Vt = q: + Σ Hierarchical o g'.
where is the number of sub sub-graphs (4 our paper). The full architecture can be found in Figure 2
The agent is trained via the Advantage Actor Critic (A2C) Mnih et al. 2016 method to maximize long term expected reward in the game in a manner otherwise unchanged from Ammanabrolu
al. (2220) (See Appendix(A.2 These attention values thus reflect the portions of the knowledge
graphs hs that the agent must facus on to best achieve this goal of maximizing reward.
Hierarchical Graph Attention Explanation. The graph attention HHeerrrchical is used to capture the relative importance of game state observations and KG entities in influencing action choice. For each sub-graph, the graph attention, #fierarchica... E foodes is summed over all the channels m to obtain a Hierarchical, E "Rdddd x1, showing the importance of the KG nodes in the ith sub-graph. The top-k k valid entities (and corresponding edges) with highest absolute value of its attention form the set of knowledge graph triplets that best locally explain the action a,-
In order to make the explanation more readable for a human reader, we further transform knowl- edge graph triplets natural language by template filling.
We create templates for each type of sub-graphs Gafr, Gatr,Ginu ;Ginu,cobb
is - "Ojject is attribute"
hal,yer,has,obje objectt have object"
jobject,in,location Object is in location"
sirationn,directionn (oriticoion.ccationn==---- ""roaaiiiiilooooooonn location 1 is in the direction of location 2", e.g. (forest, nor thousee converted FFoest in the north of house"
More examples can be found in Appendix A.4 A.4
3.2 Temporally Extended Explanations
Graph attention tells us which entities in the KG are attended to when making a decision, but not enough alone for explaining "why" actions are the right ones in the context of fulfilling dependencies that may potentially be unrewarded by the game-especially given the fact that there are potentially multiple ways of achieving the overall task. HEX-RL thus saves trajectories for hundreds of test time rollouts of the games, performed once a policy has been trained (Table I Appendix A.5) The game trajectories consist all the game states, actions taken, predicted critic values, game scores, the knowledge graphs, and the immediate step level explanations generated as previously described. HEX-RL produces a temporal explanation by performing a post-hor analysis on these game trajectories. The agent then analyzes and filters these trajectories in an attempt to find the subset of states that are most crucial to achieving the task as summarized in Figure then using that subset of states generate temporal trajectory level explanations.
Bayesian State Filter. We first train a Bayesian model to predict the conditional probability (AA) B;) of game step (A) given any other possible game step (B;) in the game trajectories. More specifically, each game step is composed of 3 elements, game state o,, action a, and the current knowledge graph G,. The key intuition here being that state, action pairs that appear in certain ordering in multiple trajectories are more likely to dependant on each other.
A linear transformation ensures that 9 R.
5

---

The set of game steps with the ( Bj) is used to explain taking the action associated with game state A. For example, "take egg" (A) required to "open egg" (B), and P(A B) 1, hence "open egg" is used as a reason whyaction "take egg" must be taken first. The initial set of game action
states is filtered into X, by working backwards from the final goal state by finding the set of states that form the most likely chain of causal dependencies that lead to it. Details can be found in Appendix A.5 and A.6
Language Model Action Filter, Following this, we apply GPT-2 (Radford et al. 2019 language model trained to generate actions based on transcripts of text games from hu- man play-throughs to further filter out impor- tant states known the Contextual Action Language Model (CALM) Yao et al. 2020). As this language model is trained on human tran- scripts, we hypothesize that it is able to further filter down the set of important states by find- ing the states that have corresponding actions that human player would be more likely to perform-thus potentially leading to more nat- ural explanations. CALM takes into observa- tion 0,, action and the following observa- tion Or+1, and predicts next valid actions 4r+1. In our work, we use CALM as filter to look for the relations between game step A and the explanation candidates Bj E X1. We feed CALM with the prompt O a, Og, to get an ac- tion candidate set. When the two game steps A and B; are highly correlated, given OA, as and OR, CALM should successfully predict an; with high probability. The game steps B,, whose associated action dg, is in this generated action candidates set, are saved the next set of filtered important candidate game states (X2).
9 Behind House
All states X
Gome Trajentorita ind (Table 1)
2 North House
PIAIB
Forest Part bi Go
GALM
@ Up Tree as big Take ego @ Up Tree
Key
X2 Xa
Input
Used Training
Location before Action
Location after Action
Bchind House
X; C X2
011
Go west
0
Kithhe
Figure 3: Temporal explanation pipeline for why the agent chose the action- --oo north" at "Be- hind House".
Semantic State-Action Filter. To better ac- count for the irregularities the puzzle like environment, we adopt a semantic filter to obtain the final important state set X3. Here, given A, B, X2, states are further filtered on the basis of whether one of these scenarios oc- curs: (1) aı and dg, contain the same entities, e.g. "take egg" and open egg". (2) GA and GB, share the same entities, e.g. "lamp" occurs in both observations. (3) A and B; occur in the same location, e.g. after taking action as. the player enters "kitchen" and B occurs in "kitchen". (4) The state has a non-zero reward or a high absolute critic value, indicating that it is either state important for achieving the goals of the game or it a state to be avoided. The final set of important game states X3 is used to synthesize post- hoc temporal explanations for why an action was performed in a particular state as seen in Figure 1 taking into account the overall context of the dependencies required to be satisfied and building on the immediate step level explanations for each given state X3. Ablation studies pin-pointing the relative contributions of the different filters are found in Section 4.4. We concluded that all three steps of the filtering process to identify important states are necessary for creating coherent temporal explanations that effectively take into account the context of the agent's goals.
Table 1: Example state saved during game play.
STEP: 16
Text Observation: Up a Tree
Beside you on the branch is small birds nest. In the birds isst large egg encrusted with jewels...
Knowledge graph: (tree, in, forest), (egg, is, interactable....
Action: take egg Immediate explanation: egg is interactable
Game Score:5
Critic Value: 5.7457

---

Table 2: Asymptotic scores games by different methods across 5 independent runs. Eps. indicates normalized scores averaged across 100 episodes of testing which occurs at the end of the training period and Max indicates the maximum score by the agent the same period. We present results on two training rewards for HEX-RL, game only and game and IM.
Experiment
LSTM-A2C
KG-A2C
SHA-KG
Q"BERT
HEX-RL
HEX-RL
Max
Game Only
Game and IM
Metric
Eps.
Max
Eps.
Max
Eps.
Max
Eps.
Max
Eps.
Max
Eps.
Max
zork1
27
31.2
34
35
33.6
34.5
35
35
29.8
40
30.2
40
350
library
8.2
14.3
15.8
16.0
19
13.8
21
30
detective
141
188
207.9
214
246.1
308
274
310
276.7
330
276.9
330
360
balances
10
10
10
10
9.8
10
10
10
10.0
10
10.0
10
51
pentari
50.4
55
50.7
56
48.2
51.3
50
56
34.6
55
44.7
60
70
ztuu
5.1
100
ludicorp
14.4
18
17.8
19
17.6
17.8
18
19
14.0
18
17.6
18
150
dcephome
1.0
1
1.0
1
300
temple
8
7.6
8
7.9
6.9
8
8.0
8
7.6
8
35
compl.
22.6
25.9
27.3
30.8
27.2
33.1
34.9
27.2
33.9
28.2
35.8
100
std dev
0.02
0.01
0.06
0.01
0.03
0.00
0.03
0.01
0.03
0.02
4 Evaluation
Our evaluation consists of three phases: (1) We show that HEX-RL has the comparable perfor- mance to state-of-an art reinforcement learning agents on text games in Section 4.1 (2) Then in Section 4.2 we evaluate our immediate attention explanation model by comparing the expla-
nations generated by HEX-RI. RI. and agents that do not use knowledge graphs (See Fig. 2 and Section 3.1. (3) Section 4.3 we compare immediate to temporal explanations, focusing on the effects that including trajectory level context when evaluating explanations in the context of agent goals. (4) In Section 4.4 we conduct human participant ablation study evaluating the individual contributions of the filtration pipeline for generating temporal explanations seen in Figure
4.1 Task Performance Evaluation
We compare HEX-RL with four strong state-of-art reinforcement learning agents- on contemporary agents that use knowledge graphs-on on an established test set of 9 games from the Jericho benchmark Hausknechh al. 2020).
LSTM-A2C is baseline that only uses natural language observations state representa- tion that is encoded with an LSTM- based policy network.
KG-A2C, Instead of training question-answering answering system like Q'BERT to build knowledge graph state representation, KG-A2C (Ammanabrolu and Hausknecht) 2020) extracts knowledge graph triplets from the text observations using a rules based approach built on OpenlE (Angeli et al. 2015).
SHA- KG is adapted from Xu et al. 2020) and uses a -baee approach to construct a knowledge graph for the agent which then fed into Hierarchical Graph Attention network as in HEX-RL. This agent separates the sub-graphs out using a rules-based approach and makes no use of any graph edge relationship information.
QEBERT. Ammanabrolu et al. (020) uses similar method of creating the knowledge graph through question answering but does not use the hierarchical graph attention architecture combined with the sub-graphs.
These baselines are all trained via the Advantage Actor Critic (A2C) Mnih et al. 2016 method- further comparisons to other contemporary agents can be found in Appendix(A. A.7 is also worth noting that most contemporary state of the art deep RL agents for text games use recurrent neural policy networks as opposed to transformer networks due to their improved performance this domain.
HEX-RL Training. trained HEX- -I on two reward types: (a) game only, which indicates that only use score obtained from the game reward. (2) game with intrinsic motivation (game
7

---

and IM,, which contains an additional intrinsic motivation reward based on knowledge graph expansion as seen in Ammmmnabroluu et al. 2020 where the agent is additionally rewarded for learning more about the world by finding new facts for knowledge graph (see Appendix A.8 A.9
and A.10
Table 2 shows the performance of HEX-RL RL and the other four baselines. We can see that designing the HEX-RL agent be inherently explainable through the use of Hierarchical Graph Attention and the sub-graphs improves the overall maximum score seen during training when compared to any the other agents. In terms of the average score seen during the final 100 episodes, HEX-RL wth intrinsic motivation outperforms all baselines with the exception of Q"BERT there HEX-RL significantly outperforms Q*BERT on one game, is outperformed on two games, and comparable on the remaining six games. HEX-RL RL thus performs comparably to other state-of off-the-an art baselines in terms of overall task performance while also boasting the additional ability to explain actions.
4.2 Immediate Explanation Evaluation
Having established that HEX-RL RL'ss performance while playing text games is comparable to other state-of-the-are agents, we attempt answer the question of exactly how useful the knowledge graph based architecture is when generating immediate ste-bbysteey explanations by comparing HEX-RL to baseline that doesn't knowledge graphs in human participant study. Two models for step-by step-by-step explanations are compared:
LSTM Attention explanations. Extracts the most important substring in the observations through LSTM attention QLSTM and then uses those words to create an explanation.
Hierarchical Graph Attention explanations. Extracts KG triplets most influenced the choice of actions by Hierarchical Attention Hierarchical and then transforming them into readable language explanations through templates.
We recruited 40 seriicppant--gnneeaan unfamiliar with the environment hand-on on a crowd sourcing platform. Each participant reads a randomly selected subset of 10 explanation pairs (drawn randomly from pool totaling 60 explanation pairs), generated by Hierarchical Graph Attention and LSTM attention explanation on three games in the Jericho benchmark: zorkl, library, and balances. We choose three games with very different structures and genres as defined in Hausknecht et al. 2020). They each require a diverse set of action types and solutions to complete and thus provide wide area of coverage when used as test beds for human evaluation of explanations. Then they are given the following metrics and asked to choose which explanation they prefer for that metric:
Confidence: This explanation makes you more confident that the agent made the right choice.
This explanation expresses more human-like thinking on the action
choice.
Understandability: This explanation makes you understand why the agent made the choice.
Variations of these questions have been used to evaluate other explainable AI systems (eg. Ehsan lt (201)). ). At least 5 participants give their preference for each explanation pair. We take the ma- jority preference from humans participants as the result. More details are shown in Appendix B.
Figure&shon shows the result of the human evaluation of attention explanations. Hierarchical graph attention explanation is preferred over LSTM attention explanation in all three dimensions. These results are statistically significant (p < 0.05) with fair inter-rater reliabilities. We also observe that these three dimensions highly, positively correlated using Spearman's Rank Order Correlation
A slightly higher proportion of participants preferred the LSTM Attention explanations in the Humann-lkennee dimension compared to the other two. The participants preferring Atten tion explanation stated that they found it intuitive but often incoherent and found the Hierarchical Graph Attention explanations to be more robotic. LSTM attention explanations substrings of the human written observation and thus have the potential to be more natural sounding than
rr3 = 0.70, p 0.01, between "confidence" and "understandabilltyy: = 0.67, p< 0.01, between "confi- dence" and "human -ikeensss"" rs 0.89, p 0.01, between likeness" and "understandability"

---

the templated Hierarchical Graph Attention explanations when they are coherent enough to be understood. The KG sacrifices small amount of likeness return for much greater overall coherence and accuracy. KGs with Hierarchical Graph Attention give us explanations that are more easily understood and inspire greater confidence in the agent's decisions.
Example qualitative LSTM Attention and Hier- archical Graph Attention explanations can be found in Appendix C.2 As our system relies on graph hierarchical graph attention gen- erate immediate explanations, knowledge graph representation module of the world knowledge is required. Most cases where the agent fails to provide satisfactory immedi- ate explanations are either when: (1) the expla- nation not directly linked to the one of the facts we choose to extract from the knowledge graph, such as object/location information; and (2) due to the error of knowledge graph extraction models themselves.
Hierarciicll
20-
Hinraclical gr grapheall
10-
LSTM
Figure 4: Human evaluation results showing the proportion of participants that prefer Hierarchi- cal Graph Attention vs. LSTM Attention explana- tions, ** indicates 0.01, + indicates k > 0.2 or fair agreement. & indicates 0.4 or moderate agreement, Evaluation results on each game are shown in Appendix
4.3 Immediate vs. Temporal Explanations
Having proved the effectiveness the knowl- edge graph the immediate step-by-ste planation level, we evaluate our method of pro- ducing temporal explanations and how they compare to the immediate explanations along two dimensions: (1) coherence; and (2) explanation accuracy when taken in the context of the agent's goals.
Participants first read trajectory the game combined with step-by-step immediate explanations and the game goal, then indicate how much they agree with the statements on Likert scale of (strong disagree) to (strong agree). Here, we add two metrics from the previous study:
Goal context: You are able to understand why the agent takes this particular sequence of actions given what you know about the goal. Readability: This explanation is easy to read.
Figure 5 shows the average scores for each question for the immediate and temporal explanations. The temporal explanations achieve comparable performance to the im- mediate explanations on all metrics except for the the metric relating to goal context where they significantly out- aut-perform the immediate explanations. A majority of participants stated that condensed trajectory level explanation made the goals of the agent easier to under- stand than reading through each step level ex- planation. These results indicate that -EX can generally successfully identify the most im- portant states in a trajectory and use them to create temporal explanations that are on par with immediate explanations in terms of herence but provide significantly more context in terms of explaining an agent's actions with respect to its task-based goals.
Apponage Rettong
Tecoul Experatoon
Cotlegey
Figure 5: Human judgment results immedi- ate and temporal explanation, * indicates < 0.05. Error bars indicate confidence level of 95%.
Cases where the agent does not provide a temporally coherent and goal-driven explanation revolve around failures particularly by the Bayesian State Filter in detecting the most important states the trajectory. qualitative analysis (as seen in Appendix C.3 suggests that this occurs cases where there are large number of branching paths that lead to the same end state. Thus, the quality of the generated temporal explanations appears to inversely proportional to the relative complexity of the game measured by its branching factor.

---

4.4 Temporal Explanation Ablation Study
Having established the overall effectiveness of the filters in HEX-RL that create the temporal explanations, we perform pair-wise ablation studies to pinpoint the relative contributions of the different filters seen Fig. We first compare explanations generated using set of important states filtered from the trajectory using the Bayes model compared to Bayes+ CALM explanation. This how applying the language model action filter affects the quality of the temporal explanations. As before, we recruited 30 participants on crowd sourcing platform. Each participant reads randomly selected subset of explanation pairs, comprised of temporal explanations filtered by Bayes and Bayes+CALM models. Figure 6a shows that after applying the CALM model to filter explanation candidates, generated explanations are significantly preferred on the "Confidence" and "Understandability" dimensions.
Similarly, we then conducted another ablation study to validate the contribution of semantic filter by comparing the Bayes+ CALM filtering method to the full HEX-RL RR using Bayes+CALM Semantii filters. The experiment setup is the same as the previous ablation study. Figure Gb shows that Bayes-CALM--Seannt performs significantly better than Bayes+CALM on all three dimensions.
CALM
We additionally observe that these three metrics are highly, positively correlated using Spearman's Rank Order Correlation both of these ablation studie When asked to justify their choices, participants indicated that the full HEX-RL system with Bayes=CALMM+Smannte filters provided temporal explanations that they felt was more understandable than alternatives. These results indicate that all three steps of the filtering process identify important states are necessary for creating coherent temporal explanations that effectively take into account the context of the agent's goals.
Dayes
Bayes
Dayes
Bayes
00
50
Fraference
60
Preference
60
30-
30-
Bayes
Bayes
10
Bayes
16-
Bayes + CALM
Humar-liksness
Understandibilty
Human- Human-lkeness= Understandabilty Category
Category
Bayes Bayes CALM explanation
(b) Bayes CALM Bayes CALM Semantic expla- nation
Figure 6: Human evaluation results on ablation study, * indicates p 0.05, indicates x 0.2 or fair agreement.
Conclusions
Explaining deep RI. policies for sequential decision making problems in natural language is sparsely studied problem despite steadily growing need. An oft given reason for this phenomenon that deep RL methods perform better without the additional burden of being explainable. In an attempt encourage work in this area, we create the Hierarchically Explainable Reinforcement Learning (HEX-R0 RL) agent which treats explainability as first-class citizen in its design by using readily interpretable knowledge graph state representation coupled with Hierarchical Graph Attention network. This agent is able to produce step- by- commentary- like immediate expla- nations and also condensed temporal trajectory level explanation via post-hoc analysis. We show that with careful design, it is possible to create inherently explainable RL agents that do not lose performance when compared to contemporary state-of-the tf-thhe-rt art agents and simultaneously are able to generate significantly higher quality explanations of actions.
4 = 0.86, 0.01, between "confidence" and "understandabilltyy.. =0779, p 0.01, between "confi- dence" and "human "artaan-iiensss"" likeness":r, 0.90, p 0.01, between "human- likeness" and "understandabiligy"
10

---

References
Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, Mikuláš Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang. Adam Trischler, and William L. Hamilton. 2020.
Learning dynamic belief graphs to generalize on text- games CoRR, abs/2002.09127.
Prithviraj Ammanabrolu and Matthew Hausknecht. 2020. Graph constrained reinforcement learn ing for natural language action spaces In International Conference on Learning Representations.
Prithviraj Ammanabrolu, Ethan Tien, Matthew Hausknecht, and Mark O Riedl. 2020. How to avoid being eaten by grue: Structured exploration strategies for textual worlds. arXiv preprint arXiv:2006.07/4 07409.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd An- nual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 344-354.
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla Asri, Mahmoud Adada, et al. 2018. Textworld: A learning environment for text-based games. In Workshop Computer Games, pages 41-75. Springer.
Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O Riedl. 2019. Auto- mated rationale generation: technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 263-274.
Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. 2021. Edge: Explaining deep reinforcement learning policies. In Thirty-Fifth Conference on Neural Information Processing Systems.
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Ale Mare-Alex Côté, and Xingdi Yuan. 2020. Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7903-7910.
Bradley Hayes and Julie A Shah. 2017. Improving robot controller transparency through au- tonomous policy explanation. 2017 12th ACM/TEE International Conference on Human- Robot Interaction (HRI, pages 303-312. IEEE.
Daniel Hein, Alexander Hentschel, Thomas Runkler, and Steffen Udluft. 2017. Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies. Engineering Applications of Artificial Intelligence, 65:87-98.
7oe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. 2019. Explainable reinforcement learning via reward decomposition. In IJCAI/ECA Workshop Explainable Artificial Intelligence.
Graham Klyne. 2004. Resource description framework (rdf): Concepts and abstract syntax. http://www. w3. orgTTH2200IREEE-----oo-------- 200400000
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: lite bert for self-supervised learning of language representations. arXiv 1942..
preprint arXiv: 1999. 1909.
Guiliang Liu, Oliver Schulte, Wang and Qingcan I.. 2018. Toward interpretable deep reinforce- ment learning with linear model u-trees. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 41 414-429. Springer.
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. 2020. Explainable reinforcement learning through causal lens. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2493-2500.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforce- ment learning. In International conference machine learning pages 1928-1937. PMLR.
11

---

Keerthiram Murugesan, Mattia Atzeni, Pushkar Shukla, Mrinmaya Sachan, Pavan Kapanipathi, and Kartik Talamadupula. 2020. Enhancing text-based reinforcement learning agents with commonsense knowledge CoRR, abs/2005.00811.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever 2019. Language models are unsupervised multitask learners.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. 2015. Policy distillation. arXiv preprint arXiv:1511.062.
Tianmin Shu, Caiming Xiong, and Richard Socher. 2017.
Hierarchical and interpretable skill acquisition in multi-task reinforcement learning, arXiv preprint arXiv:1112.072994
Kurt Shuster, Jack Urbanek, Emily Dinan, Arthur Szlam, and Jason Weston. 2020. Deploying lifelong open-domain dialogue learning. arXiv preprint arXiv:2008.08076
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. 7. Graph attention networks. arXiv preprint arXiv:1710.10903
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. 2018. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pages 5045 5054. PMLR.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang. and Xiaoou Tang. 2017. Residual attention network for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156-3164.
Sarah Wiegreffe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Thirty-fifh fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR.
Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, Joey Tianyi Zhou, and Chengqi Zhang, 2020. Deep reinforcement learning with stacked hierarchical attention for bext- games. Advances in Neural Information Processing Systems, 33.
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. 2020. Keep CALM and explore: Language models for action generation in text- text-based games. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8736-8755 Online. Association for Computational Linguistics.
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4651-4659.
Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. 2016. Graying the black box: Understanding dqns. In International Conference on Machine Learning, pages 1899-1908. PMLR.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. 2018 Learn what not to learn: Action elimination with deep reinforcement learning, Advances in neural information processing systems, 31.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main
experimental results (either the supplemental material or as a URL)? [Yes)
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g. code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL?
[Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're /urntg/uraiing? [Yes]
(e) Did you discuss whether the data you are using/ curating contains personally identi- fiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes)
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [Yes]
13


Reviews:
[{"review_id": "xnyJuKV5UI", "reviewer": "Reviewer_SZnA", "summary": "This paper explores how to leverage graph knowledge within an RL agent to provide an a-posteriori explanation of the agent's actions.\nThe authors first motivate their approach through a well-written introduction clearly stating the research objective.\nThey then detail the underlying neural mechanisms and the different filtering layers to post-process the graph and generate the language explanation.\nFinally, the authors provide two qualitative analyses: a performance benchmark on the agent abilities and a human survey on the quality of the explanation.", "questions": " I have four main concerns:\n A) the lack of model ablation\n B) the lack of discussion about the method limitation (which may be linked to ablation)\n C) the lack of qualitative results to give a better intuition of the method\n D) the lack of information for reproducibility\n\nA) The model is based on multiple design choices, yet none of this choice is really discussed. First, both the agent performance and explainability skills are based on the structure of the graph. Therefore, changing the graph, e.g., hiding/fusing subgraphs, or injecting noise, would be of interest to see how the model depends on the expert choice. It exists a discussion in Appendix C (the impact of the filtering), which is never mentioned in the paper, I would recommend discussing it in the core paper. The impact of some hyperparameters (e.g., l193, trajectories) are also not discussed while they may be of paramount importance. Finally, in 4.2, intermediate baseline models, LSTM + slot-filling without KG would allow to see whether the model is really more understandable with a KG. Here, it is a bit unfair to compare an unconstraint LSTM model with a grammar-based model while assessing for clarity. \n\nB) It may be the most significant paper weakness: positivism. In other words, the model limitations are never discussed, and the analysis remains purely quantitive without any hints for future direction. How hard is the training/tuning? What are the model mistakes? Would it be possible to compute some language statistics? And importantly, how satisfactory are the explanations? So far, the authors only provide an absolute goal score. However, I would tend to think that the current definition may be too light to apprehend the model performance correctly. Something like: would you be able to reconstruct the sequence of actions given the explanation? Are you satisfied with the current explanation? Furthermore, I would also split the analysis into successful/unsuccessful trajectories to better disentangle the soundness and correctness of the explanation. A perfect example would be: the agent was wrong, yet, did its choices make sense? In any case, happy to further discuss this point!\n\nC) So far, the reader can't have any intuition on the model skills. The only actual example is in the user interface in the appendix. Please provide some [5+] cherry-picked text-game trajectories (positive and negative!). This is very useful when supporting l367, for instance.\n\nD) Please add a table with all model parameters in the appendix and [optional] a training curve. I just cannot accept the paper without such things. For instance, the paper does not even provide the discount factor, which is not acceptable from an RL perspective. Please also add more content on the game description in the appendix, e.g. for each game, graph statistics, game interest description etc. Please add the std in Tab2; max is less interesting. \n\nSo again, the paper has many merits. I am leaning toward acceptance despite some potential improvement. However, I cannot accept it without having C and D solved and having a conversation on the model limitation.", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First of all, I want to felicitate the authors for the quality of the writing and their efforts in explaining the model and research direction. Independently of the paper's content, it was a pleasant and easy read. \nTherefore, the research direction is well-stated. I also appreciated the absence of over-claiming in the paper... except for the title, which incorrectly mentions natural language; while it is templated language. Please remove the word 'natural' from the title.  \nThe related work is a bit short, and I would recommend adding a small paragraph on the different research directions in text games that have been pursued, e.g., continual learning [1], action pruning [2], and many others.\nSection 3 gives a good understanding of the method. Although it requires a bit of engineering, they are consistent, and the method remains a promising proof of concept. I would recommend adding some module names in Figure2 (middle) in addition to the network names; otherwise, it is hard to follow. I have a few concerns and questions about the model, which I will detail later.\nIn the experimental section, the authors correctly split the performance and the explainability. The human evaluation seems to have been correctly performed, which is not too common in the ML literature.\nOverall, I got convinced by the approaches, and despite some task-specific design (especially in the filter choice and in the knowledge subgraph), the method seems extendable to other settings.\n\nOverall, I am quite positive about the paper, and it is solid enough to be accepted. However, there are still some critical points I would like to discuss which restrain me from giving a high score.\n\n\n[1] Shuster, Kurt, et al. \"Deploying lifelong open-domain dialogue learning.\" arXiv preprint arXiv:2008.08076 (2020).\n[2] Zahavy, Tom, et al. \"Learn what not to learn: Action elimination with deep reinforcement learning.\" Advances in neural information processing systems 31 (2018).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zcw2eb0S_Cn", "reviewer": "Reviewer_H13U", "summary": "This paper focuses on learning to play text adventure games using RL while producing immediate step by step explanation, as well as a trajectory level explanation using a KG based state representation combined with Hierarchical Graph Attention, which they dubbed as HEX-RL. Experiments show comparable performance for playing text-based games compared to SOTA while providing improvement in terms of explainability compared to baseline. ", "questions": "1. What is the full action space? Some discussion about it in main paper would be helpful, Also how this method can be applied in settings with large action space. ", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros: \n\n* Paper is very well written and motivated. \n* Authors provide good review of text games' literature.\n* Human evaluation and the metic for comparing intermediate vs trajectory comparison (i.e., Goal context) seem solid and interesting. \n\nCons:\n\n* The novelty is a bit low. The whole ideas used here for immediate explanation and trajectory level explanation are not totally new and novelty is limited. \nIn terms of immediate explanation, the idea of using Hierarchical Graph Attention for explainability is not new and a similar approach has been proposed in [Xu et al.]. Same applies In terms of providing trajectory level explanation. Even though there are some novelty in combining three steps filtering approach, but each of them separately is not novel.  Also authors provide an ablation study  in Appendix C which was useful experiment, but I am not fully convinced if those three filters are all necessary. I think it would have been interesting to see the performance of just a larger language model (i.e., a larger CALM) for filtering or a fine tuned version to further simplify the design of filtering. \n* I am not fully convinced of this claim in the introduction-- \"study against strong baselines that shows that our agent generates significantly improved explanations\"; would have been nice to see other closely related baseline for Immediate explanation Evaluation such as [Xu et al.]\n\nAdmittedly, I see novelty in combining two papers [Xu et al. + Ammanabrolu et al.] but I think overall novelty is limitted. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6K5l-Pi7ejF", "reviewer": "Reviewer_Jzdf", "summary": "This paper addresses the problem of generating intermediate natural language explanations for sequential decision making in IF games. The focus is to generate immediate explanations for every step in the game and to generate temporally extended explanations to justify long-term task decisions, which depend on several intermediate steps. The approach is based on a RL Framework which uses a knowledge graph special for IF games as state representation together with several attention mechanism. To generate an immediate explanation, two steps of attention first identify the most relevant subgraph and then the most relevant KG nodes. Temporally extended explanations are generated by a cascade of filtering steps to reduce the number of relevant candidate game steps. The approach is evaluated using user studies. ", "questions": "What would be required to generalize this approach to a related usecase, such as QA or Task-oriented Dialogue systems?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper provides evidence that temporally-extended explanations deliver value for generating explanations.\n\n\nMain Concern: Generation of temporally-extended explanations\n- The generation of temporally extended explanations consists of a cascade of different components, either straightfoward statistics or prior work. It is impossible to assess the impact and rationale of each component. I encourage the authors to provide an ablation study. Using user study results, I am wondering if there is not a more general solution to assess the importance of relevant game steps to form a temporally-extended explanations. For instance, attending over all game steps to estimate P(A|B) and finetuning CALM. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept