Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper addresses the problem of generating intermediate natural language explanations for sequential decision making in IF games. The focus is to generate immediate explanations for every step in the game and to generate temporally extended explanations to justify long-term task decisions, which depend on several intermediate steps. The approach is based on a RL Framework which uses a knowledge graph special for IF games as state representation together with several attention mechanism. To generate an immediate explanation, two steps of attention first identify the most relevant subgraph and then the most relevant KG nodes. Temporally extended explanations are generated by a cascade of filtering steps to reduce the number of relevant candidate game steps. The approach is evaluated using user studies. 
strengths_and_weaknesses: Strengths:
- The paper provides evidence that temporally-extended explanations deliver value for generating explanations.


Main Concern: Generation of temporally-extended explanations
- The generation of temporally extended explanations consists of a cascade of different components, either straightfoward statistics or prior work. It is impossible to assess the impact and rationale of each component. I encourage the authors to provide an ablation study. Using user study results, I am wondering if there is not a more general solution to assess the importance of relevant game steps to form a temporally-extended explanations. For instance, attending over all game steps to estimate P(A|B) and finetuning CALM. 
questions: What would be required to generalize this approach to a related usecase, such as QA or Task-oriented Dialogue systems?
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper explores how to leverage graph knowledge within an RL agent to provide an a-posteriori explanation of the agent's actions.
The authors first motivate their approach through a well-written introduction clearly stating the research objective.
They then detail the underlying neural mechanisms and the different filtering layers to post-process the graph and generate the language explanation.
Finally, the authors provide two qualitative analyses: a performance benchmark on the agent abilities and a human survey on the quality of the explanation.
strengths_and_weaknesses: First of all, I want to felicitate the authors for the quality of the writing and their efforts in explaining the model and research direction. Independently of the paper's content, it was a pleasant and easy read. 
Therefore, the research direction is well-stated. I also appreciated the absence of over-claiming in the paper... except for the title, which incorrectly mentions natural language; while it is templated language. Please remove the word 'natural' from the title.  
The related work is a bit short, and I would recommend adding a small paragraph on the different research directions in text games that have been pursued, e.g., continual learning [1], action pruning [2], and many others.
Section 3 gives a good understanding of the method. Although it requires a bit of engineering, they are consistent, and the method remains a promising proof of concept. I would recommend adding some module names in Figure2 (middle) in addition to the network names; otherwise, it is hard to follow. I have a few concerns and questions about the model, which I will detail later.
In the experimental section, the authors correctly split the performance and the explainability. The human evaluation seems to have been correctly performed, which is not too common in the ML literature.
Overall, I got convinced by the approaches, and despite some task-specific design (especially in the filter choice and in the knowledge subgraph), the method seems extendable to other settings.

Overall, I am quite positive about the paper, and it is solid enough to be accepted. However, there are still some critical points I would like to discuss which restrain me from giving a high score.


[1] Shuster, Kurt, et al. "Deploying lifelong open-domain dialogue learning." arXiv preprint arXiv:2008.08076 (2020).
[2] Zahavy, Tom, et al. "Learn what not to learn: Action elimination with deep reinforcement learning." Advances in neural information processing systems 31 (2018).
questions:  I have four main concerns:
 A) the lack of model ablation
 B) the lack of discussion about the method limitation (which may be linked to ablation)
 C) the lack of qualitative results to give a better intuition of the method
 D) the lack of information for reproducibility

A) The model is based on multiple design choices, yet none of this choice is really discussed. First, both the agent performance and explainability skills are based on the structure of the graph. Therefore, changing the graph, e.g., hiding/fusing subgraphs, or injecting noise, would be of interest to see how the model depends on the expert choice. It exists a discussion in Appendix C (the impact of the filtering), which is never mentioned in the paper, I would recommend discussing it in the core paper. The impact of some hyperparameters (e.g., l193, trajectories) are also not discussed while they may be of paramount importance. Finally, in 4.2, intermediate baseline models, LSTM + slot-filling without KG would allow to see whether the model is really more understandable with a KG. Here, it is a bit unfair to compare an unconstraint LSTM model with a grammar-based model while assessing for clarity. 

B) It may be the most significant paper weakness: positivism. In other words, the model limitations are never discussed, and the analysis remains purely quantitive without any hints for future direction. How hard is the training/tuning? What are the model mistakes? Would it be possible to compute some language statistics? And importantly, how satisfactory are the explanations? So far, the authors only provide an absolute goal score. However, I would tend to think that the current definition may be too light to apprehend the model performance correctly. Something like: would you be able to reconstruct the sequence of actions given the explanation? Are you satisfied with the current explanation? Furthermore, I would also split the analysis into successful/unsuccessful trajectories to better disentangle the soundness and correctness of the explanation. A perfect example would be: the agent was wrong, yet, did its choices make sense? In any case, happy to further discuss this point!

C) So far, the reader can't have any intuition on the model skills. The only actual example is in the user interface in the appendix. Please provide some [5+] cherry-picked text-game trajectories (positive and negative!). This is very useful when supporting l367, for instance.

D) Please add a table with all model parameters in the appendix and [optional] a training curve. I just cannot accept the paper without such things. For instance, the paper does not even provide the discount factor, which is not acceptable from an RL perspective. Please also add more content on the game description in the appendix, e.g. for each game, graph statistics, game interest description etc. Please add the std in Tab2; max is less interesting. 

So again, the paper has many merits. I am leaning toward acceptance despite some potential improvement. However, I cannot accept it without having C and D solved and having a conversation on the model limitation.
limitations: n/a
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper focuses on learning to play text adventure games using RL while producing immediate step by step explanation, as well as a trajectory level explanation using a KG based state representation combined with Hierarchical Graph Attention, which they dubbed as HEX-RL. Experiments show comparable performance for playing text-based games compared to SOTA while providing improvement in terms of explainability compared to baseline. 
strengths_and_weaknesses: Pros: 

* Paper is very well written and motivated. 
* Authors provide good review of text games' literature.
* Human evaluation and the metic for comparing intermediate vs trajectory comparison (i.e., Goal context) seem solid and interesting. 

Cons:

* The novelty is a bit low. The whole ideas used here for immediate explanation and trajectory level explanation are not totally new and novelty is limited. 
In terms of immediate explanation, the idea of using Hierarchical Graph Attention for explainability is not new and a similar approach has been proposed in [Xu et al.]. Same applies In terms of providing trajectory level explanation. Even though there are some novelty in combining three steps filtering approach, but each of them separately is not novel.  Also authors provide an ablation study  in Appendix C which was useful experiment, but I am not fully convinced if those three filters are all necessary. I think it would have been interesting to see the performance of just a larger language model (i.e., a larger CALM) for filtering or a fine tuned version to further simplify the design of filtering. 
* I am not fully convinced of this claim in the introduction-- "study against strong baselines that shows that our agent generates significantly improved explanations"; would have been nice to see other closely related baseline for Immediate explanation Evaluation such as [Xu et al.]

Admittedly, I see novelty in combining two papers [Xu et al. + Ammanabrolu et al.] but I think overall novelty is limitted. 
questions: 1. What is the full action space? Some discussion about it in main paper would be helpful, Also how this method can be applied in settings with large action space. 
limitations: NA
review_title: null

Meta-review: metareview: The paper proposes a hierarchical approach to explainable RL which combines different modules, including a knowledge graph, to generate natural language explanations. 

There has been a debate between the reviewers about this approach being novel or not which was the main concern left after the rebuttal phase. Other concerns were indeed fairly well addressed by the authors. 

Although each module in the proposed approach is not novel, it seems that the way they are used to address the specific problem of explainability and especially in text games is novel and sound. The results are convincing and the evaluation against a large number of baselines is the result of a large amount of work and a solid scientific method. 

For these reasons, acceptance is recommended. 
confidence: Certain
Decision: Accept