Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method for deriving counterfactual explanations of model decisions (relative to the correct decision the agent should have made). The general method is that by performing gradient descent over the model parameters until its decision changes, a set of properties is identified as important features which the agent had mistakenly used to make its decision. To ensure that a minimum number of properties is identified (only the important ones), a second pass of gradient descent is used fixing all but these properties to ensure they still result in a change in the decision. The method is evaluated by training a model using these explanations.
main_review: Originality: I am not intimately familiar with work regarding explainable systems / generated explanations, so it is difficult for me to place this in relation to prior work. However, the proposed method seems original relative to the related work described in the paper.

Quality: The method seems well-supported theoretically and in the simulated experiments. I would be interested to see performance without the supervised auxiliary loss. Also, I would have been interested to see results using human judgments, although prior work has found these problematically subjective. This is  not within the scope of this work, but it is surprising that there is not a reliable way to evaluate how useful explanations are to users of a system.

The authors describe several limitations to the approach, which I appreciated. I was somewhat confused about the space of high-level actions: it seems that (from Sections 2 and 7) taking a single high-level action would fully reveal the remaining environment accessible via taking that action. I am somewhat confused why this is the case (if I am understanding correctly), and what challenges would need to be addressed to remove this assumption. Similarly, I was curious about the topologies of the maps: do they have cycles? Multiple paths to a goal? How many high-level actions are available in the start state?

I appreciated the care put into the design of the method, including masked gradient descent.

Clarity: There were several points which I found confusing. I would have preferred a bit more formalization of the task itself and the learning setup. And a few questions below:
- What clues does the agent have in the university environment that one location might lead to the goal? In the other environment, it seems green paths would be the clue.
- The explanations explain a *mistake* the agent makes, conditioned on the correct action to take. However, they don't seem to explain the initial prediction made by the agent, conditioned on the image input. They also don't seem to explain why the changes in the subgoal properties result in the change in the predicted action. Is my understanding correct? (Section 7 seems to confirm this, but I'm not sure.)
- What does the space of subgoal properties look like? Does the agent have true access to these when navigating or is this something it is estimating?
- I was confused about how learning from explanations worked in Section 4. Is this just that upon a mistake, the gradient descent procedure is carried out, and the resulting parameters are used in the next environment rollout?
- Typo in L175: "boundary"

Significance: Some of the limitations discussed in Section 7 make me wonder about this method's applicability to new environments and action spaces; however, the general idea behind this proposed method seems like it could be extended.
limitations_and_societal_impact: I think the limitations of the approach were well addressed. One minor limitation I was curious about is how this approach might extend to environments that are dynamic, or action spaces which including manipulating an environment.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: - This paper discusses the challenges of providing high-quality explanations of agent navigation decision making in partially-observable environments.  PointGoal in particular is studied.
- The authors proposed a transparency-by-design method based on subgoal-based actions.  The agent selects a subgoal in the form of an area to explore next and selection is based on simple criteria.
- Given an alternative subgoal, an explanation can be generated for why the agent selected its chosen subgoal instead of alternative subgoal.
- The efficacy of the explanations are validated by showing the performance of the agent.  Since the explanations are directly part of the agent's action selection process, showing that the agent performs well also ensures that the explanations are high-quality.  The agent performs well and training via explanation helps.
- Finally, the authors show that the agent can be provided interventions to improve performance

main_review: Post Rebuttal
-----------------

I thank the authors for their response. I have increased my score to a 7 to indicate that I think the paper should be presented at NeurIPS. 

One comment: SPL is still very useful when success is nearly 100% as then it functions as a normalized version of Average Dist. In my opinion, SPL should be reported in addition to Average Dist. as this is a metric that readers from the visual navigation community will be highly familiar with, thereby improving the readability of the paper.

Original Review
---------------------
- Strengths
    - This paper approaches a difficult topic, explanations for visual navigation agents, and provides useful and insightful discussion on the topic
    - The agent action selection procedure and explanation procedure are both well-thought-out
    - The paper is very well-written and easy to follow
    - The interventions work well
- Weaknesses
    - Are the explanations better than "$$a_t$$ was selected because the agent believed it was of lowest cost" in practice?  They provided more detail, but the explanation are based on estimations from a neural network, not wrt inputs themselves.  In other words, while the actual action selection mechanism may be interpretable and transparent, the inputs to that mechanism are not.  I do not see why a human auditor would be able to determine why (using the example in Fig 1) Subgoal 1 has a 86% chance of success instead of a 94% chance.
        - The authors do note this issue and note that work on NN's interpretability may be able to help, but I am unsure if those methods are able to generate explanations in this situation since the NN is tasked with both modeling the low-level controllers ability to "explore beyond a frontier" and the input to it is more indirect than in say classification -- the network is given an input image and tasked with using that to infer properties about the rest of the environment to then estimate whether or the agent will succeed.
        - A proof of concept experiment that existing CNN interpretability techniques provide something reasonable here would greatly strengthen this paper.
    - Non-standard evaluation procedure and dataset.  
        - The datasets used for evaluation aren't standard for PointGoal Navigation.  Ideally the datasets from Gibson or Habitat should be used as these would allow for comparison with many other methods (albeit non-explainable ones).  
        - The evaluation metric of Average Dist. is also not standard nor is it explained in text.  Adding in metrics such as Success and SPL (https://arxiv.org/abs/1807.06757) would improve the paper.  
        - It is never stated if there is a train/test split or if there is no generalization be tested
        - Will the Guided Maze and University Buildings datasets be released?
    - Questions:
        - How is "explore beyond a frontier" performed?  That seems very ill-defined.
- Overall
    - While I see issues with this work, I don't necessarily think they should block publication as I do believe others may find this work useful and insightful.  I look forward to discussing this work with other reviewers.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a method that generates counterfactual explanations for navigation agents. The method is based on the "learning over subgoals" framework of [1], where each subgoal action indicates a plan and is formed by three properties: (i) the probability of success, (ii) the cost of success, (iii) the cost of failure. The method computes a subgoal's properties changes that can cause the agent to prefer another subgoal. These changes are computed by applying gradient descent onto the agent's model until the agent's preference switches. An explanation is then generated based on the changes, using a rule-based grammar. 

The authors also propose a training-based evaluation scheme for the generated explanations. Essentially, they show that training with the gradients obtained during explanation generation leads to performance improvement. 

[1] Learning over subgoals for efficient navigation of structured, unknown environments. http://proceedings.mlr.press/v87/stein18a/stein18a.pdf


main_review: **Originality**: The specific method is novel, even though it shares spirit with a lot of previous work (see related work section). To add to the list of related work, closely related ideas have recently been proposed in NLP [1, 2, 3]. The uniqueness of this work is to formulate the method in the context of partially-observable environment, and to specifically target navigation agents. 

[1] Contrastive Explanations for Model Interpretability. https://arxiv.org/pdf/2103.01378.pdf

[2] Counterfactual Story Reasoning and Generation. https://aclanthology.org/2020.emnlp-main.58.pdf

[3] Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning. https://aclanthology.org/2020.emnlp-main.58.pdf

**Quality**: 

The method for generating explanations is technically sound. It is developed on top of a well-established framework. The authors state clearly upfront the criteria that they desire from explanations and argue why their explanations satisfy these criteria. 

My main concern about quality is the training-based evaluation scheme. To me, the ultimate goal of explanations is to communicate intent of agents to humans. The criteria set by the authors also strongly emphasize the human-helpfulness and human-interpretability aspects. The training-based evaluation cannot assess their aspects and thus may not be used to replace human evaluation. The scheme mainly focuses mainly on how the agent helps itself, rather than how it helps humans obtain a better understanding of its decisions. To the latter case to be true, explanations should influence decisions or knowledge of humans. This is not happening in the scheme, as the oracle (or human auditor) simply puts out a ground-truth action without consulting an explanation.

Subjectively, I find the explanation wording quite convoluted. For example, the first sentence in Figure 3  talks about why 0 is preferred over subgoal 10 but gives a reason that talks about subgoal 11. It may not be trivial to a regular user why subgoal 11 should influence the preference between subgoal 10 and subgoal 0. Human evaluation is essential to assess the interpretability of these explanations. Such a study could motivate better ways to present the explanations (e.g. using visual cues).    

Similarly, it is not clear that the intervention experiment (line 321) is showing the role that the explanations are supposed to play. Here, the decisions on when to intervene are not influenced by the explanations. They are simply rule-based (i.e. choosing the first wrong actions). A more interesting that I would suggest is to let a human observe the explanations and decide when to intervene depending on whether the explanation is correct or not. For example, if the agent is not choosing a subgoal for a wrong reason (e.g. it is miscalculating its properties), the humans can give the correct reason, helping the agent alter its decision. 

A concern about the results: in Figure 2 top table, in the third row, the agent should also be trained with the supervised learning loss but its performance is not much better than the non-learned baseline. I am surprised that the supervised learning loss does not help at all. The L_comp is by nature also a kind of supervised learning loss but it boosts performance dramatically. What makes the difference? Could you explain this result?

**Clarity**:

Overall, the paper is easy to follow. I really like the structure of the paper, where the clear goal is set upfront. There are a few points that need clarification (which I also request in the rebuttal):
- Description of the problem: since there are many types of navigation tasks in the literature, which one is studied in this paper? How are the navigation goals expressed (by language, image, x-y coordinates, distance to goal?)? How do you generate these goals (is there a static dataset?)? What are the training and testing conditions? Are the agent trained and tested on the same or different goals?
- Description of the environment: what is given as the state of the agent (the map, the first-person view, or both?)?What are the actions of the agent? in addition to the high-level subgoals, does the agent have low-level actions? or does it simply teleport between subgoals?
- The image of the plans are hard to understand because the annotations are not explained. What do G, R, the numbers, and the letters, the colors represent? In figure 1(2), assigning letters to subgoals while referring to them using numbers is confusing. Overall, I couldn't not understand what plans are being depicted by these figures. 
- The "Quantitative Interventions via Explanations" section is missing important details. What does "accept the changes" mean mathematically? Does that mean you update the agent's model using the explanation gradient? Importantly, does the update only last during a current time step (and the agent is reset to the old parameters) or does it permanently change the agent then on? What are the changes here (from the agent's decisions to the oracle's decisions?)?

**Significance**: The paper is an attempt to extend existing explanation techniques to partially-observed navigation tasks. However, additional evaluation is required to demonstrate the effectiveness of the generated explanations in communicating the agent's intent to humans. 
limitations_and_societal_impact: I suggest the authors tailor the presentation of the explanations to make them more human-friendly. Evaluating with humans is essential to keep track of the progress. The authors mention about potentially biases when evaluating with humans. However, instead avoiding human evaluation completely, the authors can also find ways to mitigate those biases through explicit instructions or obtaining a diverse pool of human judges.
review_title: null

Meta-review: The paper proposes a method for generating counterfactual explanations for the decisions of a high-level planner that navigates an agent through a partially known environment. The planner reasons over subgoals and the proposed method performs gradient descent over the parameters of these subgoals to identify changes that cause the planner to choose one subgoal over another. A subsequent round of gradient descent is performed with all but one parameter fixed in an effort to filter out false positives. The results are converted to a textual explanation using a rule-based grammar. The quality of the explanations is evaluated by using them to train the navigational agent following much the same procedure that is used to identify the explanations.

The problem of generating a valid and concise explanation for why a policy/planner chose one action over another is challenging and touches on broader problems in interpretability/explainability that are of interest to many in the Ml community. The reviewers agree that the proposed approach is carefully designed and is technically sound. The method is presented clearly and the paper, as a whole, is well written. The experimental evaluation provides a compelling demonstration of the utility of the resulting explanations. As the reviewers note, while this evaluation is deliberately not subjective, a human evaluation of the explanations would provide more insight into the method's broader utility. However, this can be left for future work.
Decision: Accept (Poster)