Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper presents a new framework and algorithm for explainable RL.  Given some limited domain information and observed behavior by an actor, the authors propose to search through a space of possible model transformations in order to find a model and associated policy that matches the observed behavior.  The transformations then represent some aspect of deviation, and serve as a partial explanation for observed behavior.  The transforms are derived from RL literature on state-space aggregation; the search algorithm involves chaining them together. Each node in the search tree must be evaluated by computing a full policy for the transformed model, which can be computationally expensive, and which the authors discuss.  The paper has one set of experimental results illustrating an implementation of the ideas.
strengths_and_weaknesses: Strengths:

+ The framework is novel
+ This is a reasonable problem to be working on
+ The ideas are straightforward, and the paper is clearly written
+ The experiment seems clear and helpful

Weaknesses:

- While I generally liked this paper, I consistently found myself wanting more. The paper is very "chatty" and does not do a good job of balancing exposition of the ideas (which are straightforward) with some sense of application or empirical evaluation (which is very, very thin).

- I ultimately felt the problem (explainable RL) is important, but that the proposed framework should have been more general. I mean this in two specific ways:

1) A more general characterization of the idea could be "search for a model and corresponding optimal policy that matches observed behavior". This subtly includes a host of ideas that involve /complexifying/ policies, instead of just /simplifying/ them.  For example, you discuss how removing a constraint from a symbolic planning problem might suggest that the agent was unaware of the constraint, but the opposite is also true - you could /add/ constraints to the problem,  to similarly better match behavior, that would suggest the agent is laboring under constraints that it doesn't need to.  People do this all the time!

2) I am generally dissatisfied by the approach because I feel that it can only explain the /difference/ between the baseline policy and the transformed policies. Suppose, for example, that the optimal set of transforms is the null set -- in other words, the agent is behaving as expected.  This does NOT imply to me that the agent's behavior is explainable!  In other words, this framework does not contribute any explainability of the base policy. In my mind, this also motivates the more general approach of searching for matching models.  

- The framework seems a bit limited to what feels like counterfactual reasoning 

- The computational cost is real.  As the space of transforms grows, the importance of searching that space effectively becomes increasingly important. You discuss how important a distance metric is, but it's unclear to me how often such a thing exists or is practical to compute.  I like the idea of somehow transferring partial solutions or bootstrapping solvers, but these seem to dance around the issue.  Ultimately, I'm lead to wonder: will this method ever be useful for real problems?

- The experiments in the paper are underhwhelming. While the single provided experiment is clear and helpful, I feel like a much stronger empirical evaluation is necessary.  I would recommend trimming the writing to make room for experiments.

questions: 
- Why do we need an observer?  Why not eliminate that idea and simply state that there is an anticipated policy?

- The claim on lines 274-275 isn't obviously true to me. Can you please explain this further? (I think I understand why you might think it's "clear", and I think you might be wrong, so it would be helpful to have it fleshed out)
limitations: I think the authors could have done a better job of explaining the limitations of their method (although I don't think they tried to hide anything).
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes the use of MDP transforms for autonomously producing explanations. The work also introduces and formally defines the RLPE problem and empirically demonstrates the performance of their proposed approach in single and multiagent environments.
strengths_and_weaknesses: The article is clear and fairly well written, and the proposed approach seems novel and original. I believe the paper tries to tackle a very relevant and important problem. My main concerns with the work (as listed later) are related to the validity of the assumptions made with respect to the quality of explanations produced by the proposed approach. 

questions: Major comments:

1. The key idea is that an observer has an anticipated policy in mind which is assumed to arise from partial knowledge of the MDP. The explanation then corresponds to an MDP transformation/sequence of MDP transformations whose solution closely matches (satisfies) the anticipated policy. Although this approach might be reasonable in some cases, it is possible that the transformation sequence found to satisfy the anticipated policy may not actually constitute a good explanation. Since there may be multiple transformation sequences whose solution would satisfy the anticipated policy, only some of these may correspond to a meaningful explanation. 

2. Regarding the distance metric criterion used: - if the anticipated policy was indeed derived from a transformed MDP whose distance to the original MDP is large, then choosing a transformation sequence with a low distance measure may not be ideal (because the anticipated policy actually corresponds to a transformed MDP with a large distance to the original). It would help to include some lines discussing what motivates the inclusion of the distance metric. Wouldn’t it be better to instead consider for example the length of the transformation sequence and select ones with lower sequence lengths (as smaller lengths might mean simpler explanations)

3. How does one choose a suitable set of transformation functions? This is probably a critical choice because a poor set of transformations may still be able to match the anticipated policy, but may not lead to meaningful explanations.

Minor comments:

1. The figure captions need to be more detailed. Eg: It is not clear what the yellow rectangle/purple ‘x’ in fig 1 mean.
2. The phrasing of lines 164-165 is confusing
3. In lines 351-355, I am curious as to why A* was used – was it because it assumes the existence of some heuristic signal which might simulate an observer’s assumptions? Why not use say standard Q learning for example?
4. It would be good to see results in more complex/high dimensional environments. The chosen environments all seem relatively simple.
5. The results only report the satisfaction ratio with the transformed MDPs. Reporting the satisfaction ratios on the original environments might better indicate how much the transformation sequence improves the satisfaction ratio relative to the original environment.
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies explanations of RL policies. The idea is quite slick. An agent executes a policy learned using RL. An observer has some expectations with respect to the behaviour of the agent (e.g. it may expect certain actions to be taken in certain states). The goal of explanations is to explain to the observer why the agent's policy does not match observer's expectations. This may explain to the observer that their expectations are incorrect since they are not compatible with the MDP. The expected behaviour may simply be unfeasible given the MDP, and the goal is to find those infeasible situations and offer them as explanations to the observer. The problem is well formulated, and the results on 12 domains show that the new method is feasible, and it can be implemented even in tasks that don't have a PDDL representation.
strengths_and_weaknesses: The paper is very clear and writing if of high quality. The authors took care to introduce all the necessary concepts, and the key terms have accurate formal definitions (e.g., definition 7 is excellent).

The main idea is innovative, and it appears original. The relevant literature is presented, and discussed in a way that allows the reader to see where this paper sits in the related work.

Example 1 is very good, and it (along with the paragraph in lines 82-94) clarifies the goals. 

The fact that the method is domain independent is a strength.

The fact that both an intuitive explanation and formal definitions are provided is a plus.

Section 5 is very competent. The authors explain the challenges and propose a feasible solution.

One weakness is that it would be good to see what the explanations were found on the 12 domains that were evaluated.
questions: The results are excellent, but it would be useful if the appendix, for example, showed a few examples of observer's expectations, and how they were addressed by the algorithm. I can imagine that the method works, but a few examples, perhaps one example per domain, would be very useful.

Lines 18-24 in the appendix explain how to deal with non-symbolic domains? Will the derivative vector always work? I am asking because perhaps there is a counterexample that could be mentioned or a theorem that could be cited that shows that there is no counterexample.

This is a strong piece of research and I don't have other questions. The paper is well written, and it was a pleasure to read it.
limitations: There is no negative societal impact in this work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work proposes an explainable Reinforcement Learning framework, which seeks to explain the discrepancies between the learned "actor" policy and an anticipated "observer" policy of a partially informed observer. This is done automatically by (an "explainer") searching 
 over an available set of (state and action) transforms and selecting (a composition) of transforms that tweak elements in the state of the policy (producing counterfactual states), such that the (re-learned) policy in the transformed MDP aligns with the expected policy. Since the authors work on a symbolic (relational) environments, the transforms can given an indication of the key elements that had previously been overlooked by the partially informed observer (which also explains the agent's behavior, hence explainable RL). The authors do an excellent job in formalizing the definitions and explaining the intuitions. Several heuristics are proposed to perform quicker searches over the set of transforms to reach perfect alignment while making minimal changes to the states/actions. The proposed framework is shown to be applicable to a variety of domains including that of single and multi-agent RL environments, and in stochastic settings. 
strengths_and_weaknesses: Strengths:
1. This work raises valid questions on explainability in (Deep) RL, which is of great significance, given the black-box nature of the policies and their usage especially in safety critical domains like autonomous driving and treatment recommendation.
2. The authors have done an excellent job in writing the paper. I thoroughly enjoyed going through the content. The explanations and intuitions are very clear.
3. The proposed framework seems to be novel and is also generally applicable (agnostic of the RL framework). Empirical evaluations are performed on both single-agent and multi-agent environments with added stochasticity.
4. The authors have submitted code for reproducibility (although I did not run the code).

Weaknesses:
1. Results: It is rather unfortunate that the authors chose to devote just half a page to the Results sections (with just one figure) and pushed the rest of the (significant and interesting) results to the appendix. The authors have devoted a lot of space to explain things that might not be required. For instance, Figure 2 is unnecessary and can be substituted with results from the Appendix. Also I would prefer detailed explanations like those in Appendix 4 in the main paper.
2. Proposed Framework: Although the authors state that they use a RL-setting where the transition function is unknown, some action transforms that they propose work on precondition or postcondition (action effect) relaxation which is readily available in planning domains but NOT in RL domains and must be learned by the policy. Moreover, the experiments only demonstrate explainability on a limited set of transforms like all-outcome determinization (see Appendix 4) and ignore the rest.
3. Comparisons with prior work: The authors state in lines 126-130 that the work by Sreedharan et. al., 2020 is similar to theirs in deterministic environments. However, there are not comparisons presented for the deterministic experiments. I wonder why.
4. The first and the second contribution seem to go hand-in-hand: To use model transforms in the explainable RL setup, one needs to define it formally. 
5. Details from the experimental section (like comparisons with previous work, model details, algorithm details) are either vaguely mentioned or are pushed to the Appendix: See Questions 3,4,5

Overall, the paper would benefit a lot from a rewrite, especially the Experiments and the Results sections. 
questions: 1. lines 82:94 - In both examples, the reason behind the anticipation gap is due to the crucial information missing in the the observer's anticipated policy. How about the other way round: the actor's policy overlooking some crucial information. In other words, is the actor's policy always optimal? If not, the authors should clarify this just to avoid confusion.

2. line 106-108: I am not fully convinced with this statement. Anticipated behavior needs some domain knowledge, no matter how general it is. Moreover, I believe that the more general an anticipated behavior is, the more number of transforms would be required to align both the policies (hence, more retraining in the transformed environments), which in some complex environments might be infeasible. 

3. line 185: Do these mapping functions modify multiple random variables X_i of the state or just one?

4. Line 327-328: I failed to understand how the authors can tell this without training the policy in the transformed environment?

5. Is exhaustive search used for all methods: Base, pre-train, pre-cluster? Or do you prune the transforms after a certain distance is reached?

Minor Comments: 
1. To my understanding, there can be different definitions of explainability: for instance, an observer might not have any anticipations, however he/she might be interested in understanding "how" (or even "why") [1] a particular decision was arrived at. In this work, the authors focus on setups where an observer already has some expected behavior in mind. It would be good if the authors contrast this definition with those of others, or explicitly state that this is just one form of explainability they are dealing with. 

2. lines 252-254: Might be good to mention this upfront to avoid confusion. As a reader, I would definitely be in suspense otherwise.

3. Scatter plots: The points should be made bigger and more distinct. Also please make the captions more detailed (like define what X-axis and Y-axis units are). As a reader, it is inconvenient to search for such details in the text.

4. Line 374: We also measured the length of the explanation (i.e. the number of atomic ... ): I couldn't find this result in the main paper. It possible I may have missed it.

line 61: "is comprised of" -> comprises
line 147-148: "where the set of states" -> where each state
line 198: s \in \phi_{-1}(\bar{s})
line 237: "as comprised of" -> as composed of

[1] Neural Logic Reinforcement Learning, Zhengyao Jiang, Shan Luo Proceedings of the 36th International Conference on Machine
limitations: The authors do propose some limitations and future work to extend their framework in the Conclusion Section. However, to my understanding, there is a key limitation that may have been overlooked:  The authors consider a Relational (symbolic) domain, hence a discretized state with limited values. In continuous domains, with the introduction of feature extraction networks, the number of elements in a state (random variables) and its corresponding values would increase, thus increasing the set of transforms required to align the two policies. Will this render the search for transforms infeasible? I understand that this might not be withing the scope of this work (since it is in the preliminary stages), however it might be good to just comment on this for future developments in this direction.
review_title: null

Meta-review: metareview: This paper is about explainable AI: explaining a black-box agent's learned behavior via how it aligns with an observers anticipated behaviour

This paper was a bit polarizing with the reviewers. First let's summarize on the agreements between the reviewers. They all agreed:
+ the problem of study is interesting and important
+ the paper was well written and engaging
+ the formalism and overall approach is unique 

The disagreements between the reviewers revolved around:
- sharpening the text and claims within (addressed via extensive author engagement)
- the availability of good transforms
- the meaningfullness of the explanations when the agent's behavior is degenerate or there is agreement with the observer
- the completeness of the experiments provided (limited space and focus compared to other parts of the paper)
- computational tractability of the approach (searching transforms and solving for optimal policies)
- applicability in real world settings (focus on discrete symbolic domains, computation again) 

As one reviewer put it "would anyone really use this approach?". Three reviewers aligned on clear reject concluding that the paper was intriguing but there were too many loose ends and open questions left to future work. Whereas the 4th reviewer found the work to be more than enough.

The AC found both sides of the argument compelling with a slight concern that more substantive ideas were required to convince the reader that the approach is applicable to high-dimensional, messy domains like pixel-based control and robotics. Indeed, symbolic domains are great for illustration purposes, but those domains are so intuitive that behavior is often interpretable by design---afterall they are toy problems designed to highlight specific agent properties. Whereas, messy domains, like autonomous driving (operating on multiple dimensions of sensors, actuators --- all at different timescales) which the paper used as motivation are the end goal, and it remains unclear if this paper can get us there.

In the end the paper is well executed and correct so should be considered for acceptance.
confidence: Less certain
Decision: Accept