Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a method for explaining DRL behavior in a robotics setting by using a graph of robot components as input.
This input choice enables LRP to identify the importance of different components. By combining values across states or actions, different explanations are generated.
strength_and_weaknesses: This work is missing qualitative evaluations of the proposed method.
Visualizations of the explanations should be secondary to evaluations.
Guesses are made as to how the agent functions / why the explanation is of a certain form (e.g., 4.1.1), but these need to be tested, at the very least.
Some of these guesses also raise concerns. For example, the authors note that "the vicinity of the two entities in the robot is the cause of the high score". If this is the case, this does not seem to be a desirable property.
This same guess is mentioned in 4.1.2. However, an explanation method that assigns scores based on known connectivity information does not provide more information to a potential user.
These same guesses do not fully match the examples shown. Continuing with example: This pattern does not hold for other adjacent entities (e.g., if this is reason for leg_left and thigh_left, then why does this not happen for leg and thigh?).

Section 4.2 attempts to "validate the correctness of the hypotheses mentioned in Section 4.1". This is a great goal, but the approach does not achieve this goal.
Most importantly, all explanations are for a different, separately trained agent. Since the proposed approach is seeking to explain a policy/agent as opposed to the task, hypotheses from 4.1 cannot be validated using a separate agent.
In addition, the results do not support the hypotheses made. This is unsurprising, though, since the explanation method used in 4.1 differs from those used in 4.2 and are for different policies (thus, different information about different behavior).

The results are also not in a qualitative form, but comments made after-the-fact based on plots. It is important to generate a hypothesis and then test it to see how often the proposed method is useful.
(The ability to draw matching conclusions from two different explanation formats is of limited use. If one is supposed to be predictive of the other, the ability to predict should be shown.)
For example, if the scores are meant to help predict the effect of occluding a feature, then the ranking correlation between the scores and occlusion outcomes can be computed.

Some comments on methodology:
- If LRP is supposed to decompose the contribution across a layer (while keeping the same total relevance within each layer), then what is the motivation for approach #2 in Section 3?
- Occluding features does not test which feature presence is important for success, but which features the agent seeks to observe. Thus, it cannot be used to test for entity importance.
- There is no clear statement about being a local or a global explanation. LRP is a local method, yet this work is closer to a global explanation via averaging of local explanations. This approach will yield odd results in more complex domains (e.g., one where two separate behaviors are executed).
clarity,_quality,_novelty_and_reproducibility: The motivation for this method is unclear.
In explainability, there is generally a use-case. An example use-case would be to allow a human to predict the next action or to predict the action if something were to change.
The last paragraph in Section 1 serves to motivate the work, but:
- The listed applications do not permit creating testable hypotheses that an explanation can help test.
- The use-case of "figure out how severe the damage is" cannot be done with this method: if the agent can solve task in either case (for example, stand up with hands or without them), then performance would not be affected; however, agent's specific actions would be greatly affected by presence of hands, so saliency methods would highlight them.
- For the use-case of "explain the adaptation process," what is the explanation? What about the adaptation process has been explained? (The same goes for "visualization for explaining the training process")
- This section lists aspects that would be useful to explain, but there is no connection between the explanation and making conclusions of the specified types. The mechanism for making the conclusions would then form the basis of tests performed later in the paper.

To make room for thorough evaluation, the explanation of LRP can be substantially reduced. It is sufficient to outline LRP at a high level.
Likewise, in Section 4, there are paragraphs that transcribe information that is already present within the plots (without making any conclusions from this information). These redundancies can be reduced to make more room.

Further Minor Comments:
- The plots are often far from the text focused on describing them. Figure 3, in particular, is two pages ahead of its text. If possible, placing figures closer to their text would help readability.
- "Saliency methods have proved to be successful in highlighting the most relevant pixels in image classification" has been brought into question. At the very least, some saliency methods have been found to be weight-agnostic.
- It is insufficient to refer to results on LRP for graph classification to support the use of LRP in this use-case.
- The definition of the policy is odd. Suggests that probability of an action in a given state can vary over time.
- top of page 7: "(bottom bar plot in Figure 2" -> "(bottom bar plot in Figure 2)"
- 4.1.2: Referring ot FetchReach-v1 in Figure 1 while Figure 1 says v2
summary_of_the_review: This work presents a method for generating explanations, but no way to draw conclusions from these explanations.
Relatedly, the method is not evaluated, so its usefulness has not been demonstrated.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes an approach to transform state space into a graph representation and create an importance mapping between state entities and action entities. This can be seen as a tool in the explainability toolbox, however in my estimation the manuscript does not demonstration how this metric could be used in application contexts mentioned in the abstract (understanding of decision-making, recovery from faults). The main result is a mapping for two MuJoCo environments and verbal analysis of whether various portions of this mapping make sense.
strength_and_weaknesses: Overall this work does not reach the level of evidence to support or address the claims and research questions postulated in the abstract and the introduction. While the aim of the work is explainability of DRL algorithms for robotics, the actual result only covers a mapping between actions and state space entities, but does not make the next step -- showing how this information can be exploited to explain DRL agent's decision-making or make the agent's behavior more explainable or predictable.

In my mind the provided mapping is a curious idea, and I like the idea of back-tracing contributions to joints and state space entities. However this offers only a small piece of the explainability puzzle and the manuscript does not provide any empirical (or theoretical) confirmation that having this tool at one's disposal would indeed help explain agent's behavior.

Did I understand correctly from description around Equation 4, that before the algorithm is trained, all of the function approximation networks are replaced by GNNs and the the standard DRL learning process is ran, but now we have GCN features instead of plain robot state vectors? If that is so, I would like to know how well does the resulting system perform after all these changes to the architecture.

I would be interested to discuss whether the use of GNN machinery is the easiest way to reach the mapping demonstrated in this work. Could we build the same mapping by using just state space saliency maps and the actions that were performed in those states? This is not a direct critique, but it seems that adding additional learning step (training a working GNN on top of the already existing learning task) makes the problem harder and the same outcome could have been achieved by analyzing the behavior statistic of the agent without the graph decomposition.

Result presented in sections 4.1.1 and 4.1.2 verbosely describe the observations presented in Figure 1, but provide little in addition to what we can already see on the Figure. Many of claims made in those sections are speculative, but are presented as facts (such as "This case highlights one of the reasons we selected the graph structure: not only do we take into account the effect of features of each entity on the decision-making process, but we also consider their position in the structure.")

I find that the manuscript lacks the mention of the final performance of the agent that was subsequently analyzed by the proposed method. There is a mention of "training to convergence", but is not specified at what level of performance that convergence happened.

To validate the applicability of the proposed method I would also need some demonstration of the proposed metric leading to a useful insight or some other kind of application.

clarity,_quality,_novelty_and_reproducibility: Clarity: The conceptual idea is clearly presented, however the details of the method and implementation remain vague, despite the abundance of formal definitions and equations. It remains hard (at least for me) to piece together what exactly is happening in which order. The overall idea is clear though.

Quality: Low to medium: the presented results are not ablated and not validated by further experiments. The obtained metrics are just reported and then discussed, but it remains unclear whether we should trust those numbers or this is just one of many outcomes we could have obtained. 

Novelty: Limited. I would consider fresh the idea to trace back the contributions of weight back to the input space in robot domain, however the approach itself is, of course, not novel.

Reproducibility: High. The amount of detail provided is sufficient to recreate the proposed system, or at least something close to it in spirit.
summary_of_the_review: I find this work lacking in many important aspects, mainly the finding only to a small per cent supporting the claims and aspirations of the paper. I am unable to judge the explanatory power of the proposed methods as no such validation is provided in the work: maybe this is an amazing metric, or maybe these number are just marginally useful statistics of a DRL policy -- at this point this remains unknown. My recommendation is to exclude this paper from the conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose to analyze the learned representation in a robotic setting by utilizing graph neural networks (GNNs). Using GNNs and Layer-wise Relevance Propagation (LRP), they represent the observations as an entity-relationship to allow us to interpret the learned policy. Finally, they evaluate their approach in two environments from MuJoCo. 
strength_and_weaknesses: - Strengths: topic very relevant for the scientific community and promising results. 
- Weaknesses: dubious technical and methodological novelty and contribution. 
clarity,_quality,_novelty_and_reproducibility: I think the paper is sufficiently clear and well-structured. The paper and the supplementary materials should be enough to allow reproducibility. 
summary_of_the_review: The main contribution and novelty of the paper is a bit unclear: there are already works dealing with explainability of DRL techniques, and LRP has already been employed before to enhance explainability in DRL and GNNs. From this point of view, it's not clear to me what is the main technical and methodological contribution of this paper. 

I think the clarity of the paper will be enhanced if a graphical representation of the overall proposal is included in a Figure. A general scheme of the proposed method, as well as more information regarding motivation and intuitions, will be very positive. In fact, it's not clear to me what an explanation exactly is in this particular scenario and how this can be effectively employed by a human user.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new approach to interpretation of deep reinforcement learning agents via layer wise relevance propagation (LRP) applied to graph neural networks used in RL. The proposed approach produces feature-importance scores for different input features to a GNN, allowing for explanations via saliency for final policies. Salience scores are computed by backpropagating from individual actions through the GNN for multiple episodes, producing scores that can then be normalized and compared. Scores are validated by ablation studies on individual features to determine their true importance.
strength_and_weaknesses: Strengths:
* Needs for explainability in DRL are well stated, and preliminaries/background descriptions of GNNs, LRP, and SAC are clearly stated.
* Ablations are conducted to empirically validated the feature importances that are identified by the proposed method, lending support to the validity of the proposed method.

Weaknesses:
* General weaknesses of saliency methods are not discussed or addressed in this work (e.g., [1]). For example, while an individual feature may have a high saliency score, applying causality or further meaning to that score is unjustified and unfounded speculation.
* Statistical tests are performed at alpha=0.05, but there does not appear to be any correction (e.g., Bonferroni correction) for the dozens of statistical tests that have been performed. It is possible that many of the results are not significant after applying a correction, but the true statistical values are not provided beyond 2 significant figures, so this cannot be confirmed. Furthermore, effect sizes should be reported if significance tests are going to be used to show whether or not the significant effect is meaningful, and to what extent it is meaningful.
* The LRP approach to post-hoc explanation for RL agents is only applied to small state spaces with very clear alignments between state-action mappings. Experiments with larger, less-structured state spaces would be more illuminating to show whether or not the discovered features are truly meaningful.
* There are no constraints on sparsity for the discovered features, resulting in several rows of the confusion matrices (Figure 1) having several high-scoring elements. Again, this does not suggest the method would scale usefully to larger domains.
* For both domains, removing certain elements of the action space is shown to improve performance, yet these joints are not identified as being exceptionally important by the proposed method.

[1] Adebayo, Julius, et al. "Sanity checks for saliency maps." Advances in neural information processing systems 31 (2018).
clarity,_quality,_novelty_and_reproducibility: The overall presentation of the work is clear, though elements of the results need further explanation (as in the statistical significance tests discussed above).

The overall novelty of the work is not particularly striking, as the primary contribution is applying an existing technique to a new style of model (applying LRP to GNNs). While this presents potentially useful or insightful takeaways for new domains or applications, the domains presented in this work do not highlight the true power of LRP for GNNs, as the state-action spaces are too small to get a sense for the potential of this technique applied to a GNN.

The technique seems to be reproducible.
summary_of_the_review: While it is interesting to see saliency methods applied to GNNs and to deep reinforcement learning in new ways, the approach proposed in this work lacks sufficient novelty or innovation to justify a strong accept without very strong experimental evidence. Unfortunately, the experiments are also lacking, with possible statistical errors and speculative explanations for feature importance scores. Without an overhaul of the discussions to either remove or substantiate causal descriptions of the feature importance and improvements to the results reporting, the paper is not yet ready for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to use graph neural networks and layer-wise relevance propagation for explaining deep reinforcement learning in robotic settings.  As pointed out by multiple reviewers and acknowledged by the authors "LRP has previously been applied to explain DRL agents' decisions in environments with visual inputs such as Atari games. It has also been applied to GNNs to explain graph classification (not reinforcement learning)", the new contribution here is to apply this technique to a new domain, i.e., robotic settings. The authors argued that the new setting brings a few differences, such as continuous action space and sensory input. It is unclear though what changes the authors have to adopt to enable LRP in the new settings, and therefore hard to gauge if the learning will be of interest to the community. Reviewers also raised concern that the work does not support or address the claims the research questions postulated in the abstract and the introduction. It is also hard to draw conclusion from the different explanation methods used in the two different environments. 
justification_for_why_not_higher_score: Limited novelty and poor execution.
Decision: Reject