Explainability of deep reinforcement learning algorithms in robotic domains by using Layer-wise Relevance Propagation
Authors: Mehran Taghian Jazi
Shotaro Miwa
Yoshihiro Mitsuka
Johannes Günther
Osmar Zaiane
Venue: Submitted to ICLR 2023 (2023)
Keywords: Explainability
Deep Reinforcement Learning
Graph Network
Layer-wise Relevance Propagation
Robotic

Abstract: A key component to the recent success of reinforcement learning is the introduction of neural networks for representation learning. Doing so allows for solving challenging problems in several domains, one of which is robotics. However, a major criticism of deep reinforcement learning (DRL) algorithms is their lack of explainability and interpretability. This problem is even exacerbated in robotics as they oftentimes cohabitate space with humans, making it imperative to be able to reason about their behaviour.
In this paper, we propose to analyze the learned representation in a robotic setting by utilizing graph neural networks. Using the graphical neural networks and Layer-wise Relevance Propagation (LRP), we represent the observations as an entity-relationship to allow us to interpret the learned policy. We evaluate our approach in two environments in MuJoCo. These two environments were delicately designed to effectively measure the value of knowledge gained by our approach to analyzing learned representations. This approach allows us to analyze not only how different parts of the observation space contribute to the decision-making process but also differentiate between policies and their differences in performance. This difference in performance also allows for reasoning about the agent's recovery from faults. These insights are key contributions to explainable deep reinforcement learning in robotic settings.

Under review as a conference paper at ICLR 2023
EXPLAINABILITY OF DEEP REINFORCEMENT LEARN- ING ALGORITHMS IN ROBOTIC DOMAINS BY USING LAYER-WISH WISE RELEVANCE PROPAGATION
Anonymous authors Paper under double-blind review
ABSTRACT
A key component to the recent success of reinforcement learning is the introduc- tion of neural networks for representation learning. Doing allows for solving challenging problems in several domains, one of which is robotics. However, a major criticism of deep reinforcement learning (DRL) algorithms is their lack of explainability and interpretability. This problem is even exacerbated in robotics as they oftentimes cohabitate space with humans, making it imperative to reason about their behaviour. In this paper, we propose to analyze the learned representation robotic setting by utilizing graph neural networks. Using the graphical neural networks and Layer-wise Relevance Propagation (LRP), we rep- resent the observations as an entity-relaiionship to allow us to interpret the learned policy. We evaluate our approach in two environments in MuJoCo. These two environments were delicately designed to effectively measure the value of knowl- edge gained by approach to analyzing learned representations. This approach allows us to analyze not only how different parts the observation space con- tribute the decision-makir makiin process but also differentiate between policies and their differences in performance. This difference in performance also allows for reasoning about the agent's 's recovery from faults. These insights are key contribu- tions to explainable deep reinforcement learning in robotic settings.
be able
INTRODUCTION
While Deep Reinforcement Learning (DRL) has shown tremendous success in domains like games, highly structured robotic settings, and other real-world domains, it is still held back by concerns over its safety and explainability. Due to DRL leveraging non-linear function approximators (i.e.. neural networks), its behaviour cannot be fully understood and anticipated. Especially in domains where DRL is deployed alongside humans, it is expected to perform as anticipated by those humans. To fully harness the potential that comes from this powerful technique, is paramount translate the internal state of DRL approaches into human-underssandable signals.
A Reinforcement Learning (RL) agent interacts with the environment to gain knowledge and learn to perform. It observes the environment, takes action accordingly, receives feedback, and updates its behavior based the feedback. This self-training ability makes RL complex learning procedure, causing many challenges in interpreting its behavior. Combining RL with the representation learning power Deep Learning (DL) models further adds to this complexity. Explaining the policy learned by black box DL model as a function approximator is one of the major challenges in interpreting DRL models. One method proposed to tackle this challenge State Representation Learning (SRL). SRL. is feature learning method that learns low-dimensions representation of the state from high- dimensional raw observations (like pixels an image) by capturing the variation the environment caused by the agent's actions. (Lesort al. 2018 Doncieux et al.) 2018 Raffin et al.) 2018 2019 Traoré et al. 2019 Doncieux et al. 2020). While SRL methods identify the most relevant features of high-dimensional observation for learning to act and compact the observation accordingly, still require highlighting the most relevant features in www-imenniionll compact observation space robotic environments.

---

Under review as a conference paper at ICLR 2023
This work aims at identifying the relevance of each entity of robot in the decision-makinn process in low-dimensional sensory input robotic environments in which observation space is as compact as possible. Compact observation means that removing any part of the observation space would lead to a drop in performance. Saliency methods have proved to be successful in highlighting the most relevant pixels in image classification (Simonyan et al. 2013: Bach et al. 2015 Zhou al.
2016 Selvaraju et al. 2017]] Zhang et al. 2018a b). and entities and relations in graph classification
(Baldassarre & Azizpour 2019 Pope et al. 2019). Some work extended the application of saliency methods from classification to RL,, focusing on environments with visual data as states. One example of this application is explaining the DRL agent's 's behavior in Atari games by visualizing its decisions
(Weitkamp et al., 2018 Greydanus et al. 2018 8 [yer et al. 2018 Huber et al. (019)). Nevertheless, saliency methods in RL have only been applied to RL problems with visual input states. In our work, saliency method is used to highlight the contribution of each part of the robot to the policy, which helps us identify the most and least contributing parts to the decision- decision-making. making.
Since the structure of robot is similar to graphs, we represent the robot's state using graphs and apply saliency technique to highlight the contribution of each part of the graph to the agent's s decisions. [Baldassarre & Azizpour (019) claim that Layer-wise Relevance Propagation (LRP) proves to be efficient in identifying the most contributing parts of a graph to a graph classification task. Based on this claim, we choose LRP as our saliency method. First, we need use graph neural networks (GNN) as function approximators in our DRL, algorithm. After the agent's performance converges, we apply LRP to identify the most contributing components of the robot to learning the task.
A robot contains some number of body parts that are connected through joints. The body and joint in the robot correspond to the node and edge in the graph, respectively. Graph representation is used decompose the robot into its entities and their relationships. This kind of representation in DRL has been previously used by Samchez-Gonzalez Gonzalez et al. (2018) and Wang et al. (2018). The LRP highlights the relevance of every action element in the output to each entity of the observation graph, creating heat map of action-entity relevance, based on which we can distinguish the most relevant parts from the least relevant ones.
Knowing the contribution of every entity in the robot to the decision-making process is highly im- portant. One application is to provide a visualization for explaining the training process, which can be done by identifying the robot's s entities contributing to learning task during the training pro- cess. To get an intuition, assume a child is learning to stand up. During the initial stages of learning, they use their hands as assistance; however, in later stages, they can stand up easily without using their hands. Therefore, during the early stages of training, the contribution score of both hands and legs would be high, while during later stages, the contribution of hands drops. Another application is during a malfunction, where part of a robot is broken. Knowing the importance of the broken part helps us figure out how severe the damage is and whether the agent can recover from that malfunction or not. This recovery can be in the form of learning a new policy from scratch for the new dynamies or transferring the policy trained in the previous dynamics. If we choose to adapt to the new dynamics after malfunction, this method can explain the adaptation process. To have better intuition, imagine the human's 's writing task. A right-handed human breaks their right hand; after that, they start using their left hand instead. In the first dynamics, the contribution of their right hand is the highest in the writing task. In contrast, in the second one, after adaptation to the new dynamics, the importance of their left hand escalates while the right hand's importance drops.
2 BACKGROUND
2.1 GRAPH NEURAL NETWORK
Graphs are tools that are used to represent structured data. A graph contains multiple entities with relationships among them. Entities and their relationships are shown using nodes and edges of the graph, respectively, which gives us flexibility in designing representation architectures of arbitrary shapes. Furthermore, this way of knowledge representation emphasizes the location of each entity relative to other entities. Graph neural networks (GNN) are neural networks that operate on graph inputs. GNNs impose constraints on relationships and interactions among entities while finding the optimal solution. In other words, they emphasize the relational inductive bias. Our GNN architec- ture and operations are according to Battaglia et al. 2018). In an input graph, there are three kinds
2

---

Under review as a conference paper at ICLR 2023
of features: 1) node feature which shows the state of the entity, 2) edge feature which illustrates the state of each relation, and 3) global feature which indicates the state of the whole system. = {ex} and V { denote the set of edge and node feature vectors respectively and denotes the global- level feature vector. A GNN has three operations for updating node, edge, and global-level features. Equation I shows the operations in GNN.
c = (ck: Vro VANa U
6 p p (E))
V = (e VI,u)
ह p (E)
u' = (e'''v,,
v' (V")
where E = [lex: Tkk m is the updated set of edges whose receiver node and r and Sy. are the receiver and sender nodes of edge k, V [v()== 1 is the set of updated nodes, E' = F. = ( rk= 8/)) J==1 is the set of all the updated edges. 6 updates edge features incorporating surrounding nodes and global-level features in the update. d" updates node features incorporating aggregated features of the surrounding edges of a node and global-level features. Finally, o" updates global-level features given the graph's ' aggregated node and edge features (Battaglia et al. 2018). A summary of the GNN operations can be seen in Appendix A
2.2 LAYER-WEL WISE RELEVANCE PROPAGATION
Originally, the LRP has been used as a visualization tool highlighting the contribution of pixels to the classification of images by neural networks. The LRP decomposes the output probability given to a specific class and pack-propagates this score to the input components so that the sum of scores in each layer is equal across layers:
f(z)
Σ dE(++1)
Σ
( R
de!
where Ry is the relevance score given to unit d of layer I and f(x) is the score of the output layer. Bach et al. (2015) proposed two methods for calculating these relevance scores and preventing them from taking unbounded values. These two methods are the aB-rule and the e-stabilized rule. As discussed by Baldassarre & Azizpour (2019). the latter rule is more robust and simple, which is why we also use the second rule. For the details about LRP, please refer to Appendix B
Intuitively, during the forward pass, the neural network emphasizes some parts of the input that contributing more to the output by giving them higher weights and activating them. LRP uses these weights and activations in each layer to propagate the output back through the network until the input layer. Hence, it can highlight the most contributing parts of the input.
2.3 DEEP REINFORCEMENT LEARNING
Our method can explain all the policies with neural networks as function approximators regardless of the type of DRI. algorithm. We selected Soft Actor-Critie (SAC), popular and known to perform well in robotic environments (Haarnoja et al. 2018ap. Our problem is a policy search in Markov decision process defined by tuple (S,A, P. r) where S and A are continuous state and action spaces, p: SxSxA + [0, oo) is the state transition probability density, and r S Sx.A- + [rmin Tmax] is the environment's reward function. The policy, denoted by is the probability of selecting action a in state & at time step f. SAC is trained through an off-policy process, meaning that the policy being evaluated and updated is separate from the policy used for generating trajectories. SAC's objective function maximizes not only the expected cumulative reward but also an entropy term. Therefore, the optimal policy looks as follows:
' arg max B(ono))- [r(se, at) aH((T((s())
where a is the temperature parameter that determines the importance of entropy, and H((T(.(st) is the entropy term. This entropy term facilitates the optimization process by smoothing out the objective function to avoid getting stuck in local optima. Empirically, this entropy term incentivizes the agent to explore the environment more widely. For the details, please refer to the original paper (Harnooj et al. 2018b). We also use automatic temperature parameter tuning, as discussed in the paper.
3

---

Under review as a conference paper at ICLR 2023
3 PROPOSED METHOD
Our method has two phases. In the first phase, we train the agent until convergence using graphs as input observations and GNNs as function approximators for the DRL algorithm. Afterward, apply the LRP algorithm to the learned policy to provide a heatmap showing the relevance between each action element and observation entity across time-steps. Then, by using the LRP scores, we identify the most critical entities and joints in the observation and action spaces, respectively. The importance scores calculated in this phase are evaluated in the second phase. Each joint plays two roles in decision-making. First, its features are given to the agent as the state. Second, an action in the form of torque is applied to that joint, creating motion. Therefore, there are two kinds of information extracted by the LRP:
1. Entity Importance in the Observation Space:
Each action element gives relevance score to each entity of the observation space. These relevance scores are averaged for an entity across actions to yield the importance of that entity to the whole decision-making process.
2. Joint Importance in the Action Space:
The relevance scores given by each action to entities of the observation space are averaged for each action element (i.e., joints) across entities to yield the importance of the joint in the action space.
Note that we use the terms "Entity" and "Joint" to denote the observation and action space elements, respectively. The reason is that not all the elements of the observation space are "Joints", unlike the elements of the action space. For each environment, the evaluation targets the two kinds of information provided above. To evaluate the importance of the entities in the observation space, we occlude the features of that entity by removing its features from the observation space and then rerun the experiments in the new (partially observable) environment. We can validate the importance score for each entity based on the drop in performance, which is expected to be proportional to the importance score given the occluded entity. For the importance of a joint in the action space, we block that joint so that no torque (motion) can be applied to it. Again, we can validate the importance scores for each joint based on the amount of drop in the performance, which is expected to be proportional to the importance score given to the blocked joint.
A high-level sketch of the problem is to identify the contribution of each component of the robot to the decision-makinn making process. The first step of our solution is to treat each part as an entity with relationships between each pair of entities, then leam a policy on the environment using this decomposition. Hence, the state 8, at time step t would be in the form of:
8t = G(V,, Fe,, uz)
(4)
where V1 and E, are the set of node and edge features time step t respectively and is the global- level feature vector. To use graph representation by the DRL, we need to replace fully-connected networks function approximators with GNNs. There are 5 networks in SAC as follows: policy network v. Q-networks 0, and 02, and target networks 0, and 0g as explained in the algorithm GNN architecture extract
(Haarnoja al. 2018b). Wherever have states as input, features from the state.
Within one layer of GNN, three operations are done in order as in equation [ 1) update edge features using Φ², 2) update node features using ø", 3) and update global features using d". In our architecture, for Φ², we avoid incorporating node or global features in the update; for d", we avoid incorporating global features in the update (see equation []. This avoidance helps us speed up the network update without any drop in performance. However, for Φ". the update is the same as mentioned in equation [] The first and second GNN layers update edge, node, and global feature vectors' sizes to 256 and 128, respectively. All the aggregation functions p are average pooling. The global feature vector of the output graph is the network's 's output. This output is concatenated with the action for the Q-networks and fed into a 3-layer fully-connected network of size 256. For the policy network, since there are output layers: one for the mean and another for the standard deviation of the Gaussian distribution, we use different ø" functions at the output layer for each one of them.
After training the agent until convergence, we proceed with the explanation phase. In this phase, there is no further update to the policy, and want analyze the learned behavior. At each
4

---

Under review as a conference paper at ICLR 2023
time-step f. given the state of the environment, the policy network outputs the action oe (s)) corresponding to state St. This action, which is the mean of the Gaussian distribution, would look like the following:
4 (1) (2) [a ag
(K)
(5)
where h is the number of action elements corresponding to the number of actuated joints. To calcu- late the relevance of each action oz where {O...., 1). the input graph components, zero out all the elements in the action vector except the element at index i, which forms the relevance
score of the action at time step r in the output graph's global features. If e; R denotes one hot vector whose elements are zero except the one at index (which equals 1), then
r
a4
This relevance score is set to the global features of the output graph, then back pack-propagated to the input. If denote the layer-wise relevance propagation operation on neural network with LRP(),
R = LRP( for c in CO....,C-1)
where C' is the number of components of the input graph, RE is the relevance of action i to com- ponent e of the input graph at time-step f. The LRP() back-propagee the vector ((() to the input graph's 's components. Then, the relevance of each action corresponding component of the input is averaged across time steps. The evaluation phase is summarized in algorithm
Algorithm Calculating relevance scores for the components the observation space.
1: let R = 0 for (O....,
... C }} and i E (O...., h 1} R is the relevance score given by action to component of the input. can be either a node, an edge, or the global unit
2: let N denote the number of episodes
3: for each episode n in {O...., N - 1 do
4: for each time-step in {0, ................................ 1} do
5: Sample current state $; of the environment
6: 0, (S²)
a, equals the mean of the policy distribution
7:
for each element of the action vector do
8:
r!!) 0t 64
R = R!! + -RRP(r)) (6)
9:
10: end for
11: end for
end for
13: return Vc, i R /T x N)
> Average of the relevance across time-steps
EXPERIMENTS
The experiments OpenAl Gym OpenAl 2021): Wa kalke22D v2 and FetchReach
two simulated robotie environments in MuJoCo Todorov al. (2012)
The observation space is converted graph with only edge and global features. The walker is a two-dimensional two- legged figure that consists of four main body parts single torso at the top (with the two legs splitting after the torso), two thighs in the middle below the torso, two legs in the bottom below the thighs, and two feet attached to the legs on which the entire body rests. The goal is to coordi- nate the torso to move forward by applying torques on the six joints connecting the six body parts OpenAl (2022). The fetch robot is a robot arm consisting of 7 joints ordered in a line, where the is the joint closest to the base, and the wrist.roll is located at the other end, connected to the end-effector. The goal is to move the end-effector to specific 3D position given by the enviromment, i.e., the goal The actions for both environments are torques applied to joints. For both Walker2D-v2 and Fet FetchReach-vl the edge feature is the position and velocity of the joint. Walker2D-o -v-2 the torso in the model a joint connecting the robot to the world, showing its position and velocity. In FetchReach--r the global feature is the goal position.
5

---

Under review as a conference paper at ICLR 2023
4.1 TRAIN AND EXPLANATION PHASE
This phase starts by training the DRL agent until convergence on the two environments, followed by applying LRP to the learned policy to generate heat map. This heat map represents relevance of each action element to each observation entity, in Figure The relevance scores for each envi- ronment are calculated and averaged across 10 seeds, and each run for 20 episodes. We can calculate contribution scores for each entity and joint by averaging LRP scores across every observation entity and action element separately 1 The entity importance score is on the top-left and the action impor- tance score is on the bottom-left of Figures 2 and 3 for Wa Walker3 kerDDv2 and Fet FetchReach-v respectively. The correctness and intuition of the following hypotheses are discussed in Section
Averager entity-action relevance score
FetchfRecc--22
Walker2d-122
choulder ga0
0:38
0.08
0.30
0.03
thigh
1.00
0.62
0.20
0.04
0.05
0.05
0.05
shostdder
1.00
0.24
0.34
0.01
0.64
0.14
0.52
0.00
part
N9.
Seeeoee 200% SCrOS 2005
Actions TRoque appteed to uppeorurte
0.63
0.29
0:11
1.00
1.00
0.00
0.00
0.01
0.11
0.03
9.02
0.10
0.30
Stigh left
foream
1.00
0.06
2.02
0.02
0.20
0.28
0.07
0.00
Megg leff
1.00
0.31
0.30
0.00
0.03
0.53
0.03
fast left
wiit
0.04
0.81
0.00
0:27
0.07
0:25
0.01
1.00
stesster 2M1
shaulder A0t
forerr rall
Emo
nhigh
Hg
Enst
unih feft
leg Ae
fooa lett
upperam
observation
Figure 1: The LRP heat map for action-entity relevance score. The y-axis and x-axis show elements of the action and entities of the observation, respectively. Since joints have different characteristics and possible amounts relevance scores for each action across all the observation entities by dividing by the maximum score given by that action.
torque, the actions have different ranges. Therefore, we normalize
4.1.1 WALKER2D-V2
Figure right represents the heat map produced for the Walker2D-v2 environment by the LRP. As the heatmap indicates, scores given to the torse entity are the highest across all the elements of the action space. The reason is that the goal of the environment to increase the speed of the torso. Another thing that expect to observe is the high relevance of the action elements their corresponding joint entities in the observation space. By ignoring the torso column from the heat map. this dependence be nicely seen the heat map in which the relevance scores around the diagonal are relatively high. For example, for the action (torque) applied to the thigh joint, the relevance score given the thigh entity (i.e., 0.62) high relative other entities. The and leg_left. For the thigh_left action, in addition
same is true for leg, thigh_lefi thigh_leff entity, the relevance score given the leg_left entity is also high. This case highlights one of the reasons we selected the graph structure: not only do we take into account the effect of features of each entity on the decision- massing process, but we also consider their position in the structure. In other words, the vicinity of the two entities in the robot is the cause of the high score given to the leg_left entity by thi thigh_left action.
According to the entity importance plot (top-left bar plot in Figure 2 the most important entity to policy is torso the goal
environment. The importance score for other entities is pretty small; however, the score of leg. thi thigh_left and leg_left are higher than the remaining.
'The importance scores not the average scores on the heat map because the scores the heat map normalized for each action.

---

Under review as a conference paper at ICLR 2023
Moreover, the action importance plot (bottom-left bar plot in Figure indicates that the critical action elements are foot and Foot.left joints, respectively.
4.1.2 FETCHREACH-VI
Figure T left shows the heat map generated by the LRP for the FetchReach-vi environment. The position of the goal receives a high score across all actions (except wrist.roll] ). This is because all the torques should be adjusted in a way to position the end-effector at the goal. Ig- noring the goal column, similar Walker2D-ve 's heat map, we can see diagonal pattern. The highest scores for shoulder.pan elbow_flex forearm_roll wrist_fles wrist.roll actions belong to their corresponding joint entities in the observation space as ex- pected. Although for shoulder 1ift and upperarm rarm roll actions the scores on the diagonal are unexpectedly not the highest, still the scores given to their corresponding joint entities in the observation space are relatively high. This unexpected result might be explained using the graph structure and the vicinity of joints: for shoulder.lif and uppers aameeramm_roo rm-roll actions, the rele- vance scores are distributed across multiple entities in the observation space.
and
According to the entity importance plot (top-left bar plot in Figure 3. the most important entity to the policy is goal Other than goal, choulder-pane forearm.rol. wrist.rol and upperarm.row have relatively high importance respectively. Other entities have pretty small importance scores. Morcover, the action importance plot (bottom-left bar plot in Figure 3 indi- cates that the critical joints for positioning the robot's end-effector to the goal are e blow flex, brist_fess and forea rrreamm_roll joints, respectively.
4.2 EXPLANATION EVALUATION PHASE
The purpose of this section is to validate the correctness of the hypotheses mentioned in Section To ensure that the contribution of each part to the decision-making does not depend on the type of neural network, we use fully-connected networks in this phase function approximators. As discussed, the evaluation targets the importance of entities in the observation and joints in the ac- tion. The first one is fulfilled by occluding the entity in the observation space, and the second by blocking the joint. In each case, their importance scores are validated based on the drop in per- formance. In Figures 2 and the upper and lower rows reflect the importance of the observation entities and the action elements evaluation results, respectively. The importance scores are normal- ized by dividing by the maximum score and the performance bars are divided by the performance of the st anda rd setting. The standard rd setting is the original environment with no occlusion in entities nor block in joints. The significancy tests are I-test with 95% confidence interval, meaning that the two performances are significantly different for p < 0.05. Also, the results for performance bars of Walker2D-yy and FetchReach-v2 are averaged across 30 and 10 seeds, respectively. The reason is that Walker2D-v2 's performance while Fetch was stable with a low variance.
unstable across seeds with high variance,
4.2.1 WALKER2D-VZ
First, we discuss the observation entity importance evaluation in the upper-row plots of Figure 2 The top-left bar chart shows that the most important entity in the observation space is the torso. with a score of 1.0. The importance bars for entities other than torso are lower than or equal to 0.3, among which leg thigh.left and leg_leff have the highest scores, By occluding the torso from the observation space, the agent could not learn policy at all. Therefore, it has the most significant drop in performance compared to the stadardd case in the upper-middle plot of Figure The performances of the thigh. leg thigg left. and leg left are equal, as can be seen in their performance bars and significancy test (i.e., their performances are not significantly different) the upper-middle and -right of the Figure 2 The foot and the foot_left have the least drop in performance compared to the standard setting, as shown in their performance bar, which can be deduced from their importance score. Nevertheless, the importance score for the thigh is unexpected because, based on its performance bar, we expect its score to be noticeable, similar to leg. thi thighh_aa gh.left....ggg.g..eee and seg_left.
The joint importance in the action space evaluation is reflected in the lower-row plots of Figure 2 As the bottom-left bar chart indicates, foot and foot left joints are the most important. Their
7

---

Under review as a conference paper at ICLR 2023
Evoluating Importance scores for Walker2d-v2 environment
Endiyy importance the observation space
Avcrage return after occuusing joines
Perforance statistical T Test
17
2:10
0.6
Aunegge Retur
foot
2:00
0.03
9.6
0.4
star dandard
3:00
0.2
thig
0.0
Entiyy
0.6
standerd
tlihh
toro
thigh joint
leg josnt
Joint Importance action
after slocking joints
Food jerd
(rigg
1.2
Foot
1.0
0.0
1:0X
0.17
standard
0.00
3:00
9:9
040
0.4
thigh
1:04
0.09
0:00
0.2
0000
Enik 65
Number of tptodds
Endty name
Figure 2: Evaluating explanation for the Walker2D-v2. Upper-left: entity importance in the observation, upper-middle: final behavior performance after occluding each entity, upper-right: sig- nificancy test for the final behavior after occlusion, lower-left: joint importance in the action, lower- middle: final behavior performance after blocking each joint, lower-right: significancy test for the final behaviors after blocking.
performance bar in the bottom-middee figure proves the correctness of this hypothesis by showing that the amount of drop in performance after blocking foot_left and foot joints is substantial compared to the st andarrd setting. For leg and leg_left the amount of drop in performance is close to each other. Both leg and 1eg left 's importance scores explain this drop; however, the leg.le le 's importance is low, although it was expected to be almost similar to leg 's score. One interesting thing is that the performance improves after blocking thigh gh and t thigh Although LRP gives a pretty low importance score to these two joints, it does not provide enough information to explain this performance improvement.
joints.
4.2.2 FRTCHREACH-VI
First, we focus analyzing the observation entity importance in the upper-row plots of Figure 3 The upfer-left bar plot indicates that the most important entity is the goal. As expected. when occluding the goal, the performance drops significantly, as indicated in the upper-middle plot. After goal, shou 1 derdddrppa. and forea airearmrrlll entities have the highest scores. Their corre- sponding performance bars show proportional amount of drop in performance after occlusion. We expect the wrist_roll joint to receive a relatively low importance score based on its performance bar. Although the wrist.roll joint seemed to be critical according in Figure [] the action applied to this joint gave
its importance score, if
we look at the heat map of the FetchReacht a relatively low score to the goal. Thus, we can conclude that this joint does not contribute to reaching the goal. That is why after occluding it, the performance did not change. The high score of wris sttrrll 1 entity is only because of the score given by the wrist_rol) action. The importance score of the upperarm_rol upperarm.roll and e elbow_fll flex performance bars are approximately equal, we expect a high score for the e bbow_flee unlike its current importance score. It remains shouldes 1 i ft and wrist_fles entities that, as clear from their performance bars, the amount of drop in their perfor- mance can imply their importance score.
entity can also be verified by its performance bar. However, since the
The joint importance in the action space evaluation is reflected in the lower-row plots of Figure 3 As indicated on the lower-left bar plot, the most critical joints the actions are e lbow flex.
8

---

Under review as a conference paper at ICLR 2023
Evaluatinn importance scores for FetchReeah--22 environment
Entity importanoe in the osservatio on soace
Average return after occluding joints
Performance statistical test
sTouddee
3.0
0.8
0.6
EndRy nS00
stcudder
0.5
X
010
StanSard
0.3
12
02
bow Fax
022
Joosibe par
standard
gool
diniider par joir
Jlou Ii1 jain
Joint Imporfance action
fveraae return after blecking
uppersr jaint
ailon jart
3.0
joint
0.8
Enily tafice
-06
Imppe
0.66
0.4
standard
M
bow Flct
2.2
Tnitiws
Number of Epteadees
SHAUDOY
EnDRY name
Figure 3: Evaluating explanation for the FetchReach-vl. Upper-left: entity importance in the observation, upper-middle: final behavior performance after occluding each entity, upper-right: significancy test for the final behavior after occlusion, lower-left: joint importance in the action, lower-middle: final behavior performance after blocking each joint, lower-right: significancy test for the final behaviors after blocking.
wrist flex. and forearm roll joints, respectively, and are highly strategic for reaching the goal. The noticeable drop in the performance after blocking these three joints, shown in their perfor- mance bar in the lower-middle plot, implies their importance. As discussed, the wrist. prist_roll joint does not contribute to reaching the goal. Therefore neither its occlusion nor its blockage affects the performance, For bar compared to the standard setting can be correctly explained by their importance score. Nev- ertheless, the LRP fails to explain the performance improvement after the shoulder .1 1 ft joint's blockage. For shoulder pan. joint, we expect that LRP gives an importance score approximately equal to the foreard rm roll 11 joint because their performance is nearly the same.
and sicuulder_li joints, the drop in their performance
CONCLUSION AND FUTURE WORK
This paper proposes a novel technique for interpreting the deep reinforcement learning algorithms in robotic domains using graph neural networks and Layer-wise Relevance Propagation. This method identifies the contribution of the robot's components to the decision-making process, al- lowing us analyze the learned behavior. The experimental results prove that our method could successfully highlight the importance of each part of the robot to decision-making. Although the contribution scores given to some entities were unexpected, for some of them, their importance could be explained by referring to the original heat map, as we did for the wrist roll in the Retchhee environment. The LRP also fails to provide additional information about the performance improvement after blocking some joints. Knowing the contribution of every part of the robot to performing task is paramount, especially in realizing how important each part is in every stage of learning task. Furthermore, during a malfunction, it can give an intuition of the sever- ity of the damage and whether adaptation is possible or not. Even after the adaptation, provides additional information about how the robot substitutes the malfunctioning part by utilizing another internal part, which can be achieved by comparing the contribution scores of the two parts before and after adaptation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wisg wise relevance propagation. PloS one, 10(7):e0130140, 2015.
Federico Baldassarre and Hossein Azizpour, Explainability techniques for graph convolutional net- works. arXiv preprint arXiv: 1905.1386, 2019.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonnzlez,, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1xXiv:1806.0220 2018.
Stephane Doncieux, David Filliat, Natalia Díaz-Rodríguez, Timothy Hospedales, Richard Duro, Alexandre Coninx, Diederik M Roijers, Benoît Girard, Nicolas Perrin, and Olivier Sigaud. Open- ended learning: conceptual framework based representational redescription. Frontiers in neurorobotics, 12:59, 2018.
Stephane Doncieux, Nicolas Bredeche, Léni Le Goff, Benoît Girard, Alexandre Coninx, Olivier Sigaud, Mehdi Khamassi, Natalia Díaz-Rodríguez, David Filliat, Timothy Hospedales, al. Dream architecture: a developmental approach to open-ended learning in robotics. arXiv preprint arXiv:2005.06223, 2020.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International Conference on Machine Learning, pp. 1792-1801. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: arXiv:/812.00 18/2.05905 2018b.
Tobias Huber, Dominik Schiller, and Elisabeth André. Enhancing explainability of deep reinforce- ment learning through selective layer-wise relevance propagation. In Joint German/Austrian Con- ference on Artificial Intelligence (Künstliche Intelligenz), pp. 188-202. Springer, 2019.
Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Transparency and explanation in deep reinforcement learning neural networks. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 144-150, 2018.
Timothée Lesort, Natalia Díaz-Rodríguez, Jean-Franois Goudou, and David Filliat. State represen- tation learning for control: An overview. Neural Nenworks, 108:379-392, 2018.
OpenAl. Getting started with gym. httpps://yym.ppeaai.coo cocc///ssacccccooc////occoc 2021. Accessed:
2022-01-29.
OpenAl. Walker2d.
walker2d/ 2022. Accessed: 2022-08-16.
Phillip E Pope. Soheil Kolouri, Mohammad Rostami, Charles E Martin, and Heiko Hoffmann. Ex- plainability methods for graph convolutional neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 10772-10781, 2019.
Antonin Raffin, Ashley Hill, René Traoré, Timothée Lesort, Natalia Rodríguzd and David Fil- liat. S-rl toolbox: Environments, datasets and evaluation metrics for state representation learning. arXiv preprint arXiv:1809.00 09369, 2018
Antonin Raffin, Ashley Hill, René Traoré, Timothée Lesort, Natalia Díaz- Radrigodrg and David Fil- liat. Decoupling feature extraction from policy learning: assessing benefits state representation learning in goal based robotics. arXiv preprint arXiv: 190100865, 2019.
10

---

Under review as a conference paper at ICLR 2023
Alvaro
Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pp. 4470-4479. PMLR, 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/1/2.6033, 2013.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
René Traoré, Hugo Aiselles-Dupré Timothée Lesort, Te Sun, Guanghang Cai, Natalia Díaz- Rodríguez, and David Filliat. Discorl: Continual reinforcement learning via policy distillation. arXiv preprini arXiv:
2019.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Representations, 2018.
Laurens Weitkamp, Elise van der Pol, and Zeynep Akata. Visual rationalizations in deep reinforce- ment learning for atari games. In Benelux Conference on Artificial Intelligence, pp. 151-165. Springer, 2018
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126 (10):1084-1102 2018a.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8827- 8836, 2018b.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
A COMPUTATION STEPS IN A GRAPH NEURAL NETWORK
The computation steps of a GN block is summarized in algorithm These steps comprise of main
updates:
1. The first part relates to edge updates. The algorithm updates edge attributes in lines 2-4 using the of function.
2. The second part is dedicated to updating node attributes (lines 5-9). In this part, first, we select the set of edges whose receiver node is node i denoted by EY (line 6). Then the edge features of the edges in F would be aggregated using pr to be used in the update function @" to update attributes of node in line 8.
3. The last part is updating global attributes (lines 10-14). V and E" denote sets of updated node and edge attributes, respectively. Firstly, the updated edge attributes aggregated using - and called r'_ Secondly, the updated node attributes are aggregated using pa -m and called v'. Then these aggregated node and edge attributes used in updating global attributes using d".
The order of steps in algorithm fistthmmisss irrelevant. One can change this order to update global attributes, per-node attributes, then per-edge attributes.
11

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Steps of computation in GNN block
function GRAPH NEURAL NETWORK(E, V, u)
2: for h E [1...N") do
3:
c/. x (Ck, Vra. Vscs u)
Compute updated edge attributes
4: end for
5:
for E No......... do
6:
let E (e Pke 8&) I mi
7:
e
= p B.)
> Aggregate edge attributes per node
8:
V; x" ( Vo, u)
> Compute updated node attributes
9:
end for
10:
let V
11:
let E' (c Mk: 9k) k=1
N
12:
e p
D. A Aggregate edge attributes globally
13:
v' (V
Aggregate node attributes globally Compute updated global attribute
14:
u' = (" v u)
return (E', Vi u')
16:
end function
B LAYER-WISE RELEVANCE PROPAGATION METHOD
The LRP method is explained according to the work by Bach et al. (2015). the relevance scores from higher layers are introduced messages sent from those layers. Therefore, the relevance of neuron i at layer 7 (except the last layer) is computed as follows:
R
Σ
R
input neuron
where (1.++1) shows the relevance coming from neuron k in layer 7 + 1 to neuron in layer 1. The relevance of the last layer defined as the classification score f(x). Equation g checks the sum of relevance scores with respect the output neurons from the input neuron. We can consider the other way and check the sum of relevance scores of the input neurons for the output neuron:
R (++1)
(.++1) R R
input
Equations 8 and are the main constrains defining LRP. Multi-layer networks are commonly built of interconnected neurons organized layer-wise. We denote neurons from layer by x; and neurons form layer + 1 by x.5. In the same manner, the summation over neurons of layers and
I ++1 1 are denoted by Σ; and Σ; respectively. A common mapping from one layer the next one consists of linear projection followed by non-linear function:
Zij KiWii
(10)
2j = Σ 24 + by
(11)
Ty g(2))
(12)
where wiy is the weight connecting neuron T; to neuron Tj. by the bias term, and g is a non- linear activation function. Common non-linear functions can be Rectified Linear Unit (ReLU) or hyperbolic tangent (lanh). One possible choice of relevance decomposition for messages from layer j to layer i is follows:
((.++)
Zij (1+1) R
Ritjj
(13)
2j
This type of formalization guarantees the conservation properties of equation 2 One drawback of the equation [3 is that for small values 2,. Ri-; can take unbounded values. Two solutions provided to overcome this drawback are e-stabilizer and a:f-stabilizer.
For the g-stabilizer, let e 2 0, then the relevance scores would as follows:
(1.+1)
Esz
R
(T+1)
2j
0
(14)
2ig
[+1)
2j
12

---

Under review as a conference paper at ICLR 2023
One problem with this method is that the relevance can be fully absorbed if the stabilizer becomes very large. For this case, we use the alternative a-S-stabilizer, which treats negative and positive pre-activations separately. Let 2J = E, bj and 2j E 2 bj be negative and positive part of pre-activation respectively, where and "_" denote the negative and positive values of žij and by. The relevance scores are then calculated follows:
all++1) (tj
R((t1)
i
+
B.
(15)
where 3 = 1. This method also control the importance of positive and negative evidence by changing the values of a and 3. The complete layer-wise relevance propagation procedure for neural networks is summarized in algorithm
Algorithm 3 Layer- wise
wise relevance propagation for neural networks
1: let ((4) fx)
2: for E {
do
3: Rll.i+11 as in equation 14 15
4: R = E, ((.++1)
5: end for
6: return vd: R
13


Reviews:
[{"review_id": "pEFJvybT2j3", "reviewer": "Reviewer_jwcH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes an approach to transform state space into a graph representation and create an importance mapping between state entities and action entities. This can be seen as a tool in the explainability toolbox, however in my estimation the manuscript does not demonstration how this metric could be used in application contexts mentioned in the abstract (understanding of decision-making, recovery from faults). The main result is a mapping for two MuJoCo environments and verbal analysis of whether various portions of this mapping make sense.", "summary_of_the_review": "I find this work lacking in many important aspects, mainly the finding only to a small per cent supporting the claims and aspirations of the paper. I am unable to judge the explanatory power of the proposed methods as no such validation is provided in the work: maybe this is an amazing metric, or maybe these number are just marginally useful statistics of a DRL policy -- at this point this remains unknown. My recommendation is to exclude this paper from the conference.", "strength_and_weaknesses": "Overall this work does not reach the level of evidence to support or address the claims and research questions postulated in the abstract and the introduction. While the aim of the work is explainability of DRL algorithms for robotics, the actual result only covers a mapping between actions and state space entities, but does not make the next step -- showing how this information can be exploited to explain DRL agent's decision-making or make the agent's behavior more explainable or predictable.\n\nIn my mind the provided mapping is a curious idea, and I like the idea of back-tracing contributions to joints and state space entities. However this offers only a small piece of the explainability puzzle and the manuscript does not provide any empirical (or theoretical) confirmation that having this tool at one's disposal would indeed help explain agent's behavior.\n\nDid I understand correctly from description around Equation 4, that before the algorithm is trained, all of the function approximation networks are replaced by GNNs and the the standard DRL learning process is ran, but now we have GCN features instead of plain robot state vectors? If that is so, I would like to know how well does the resulting system perform after all these changes to the architecture.\n\nI would be interested to discuss whether the use of GNN machinery is the easiest way to reach the mapping demonstrated in this work. Could we build the same mapping by using just state space saliency maps and the actions that were performed in those states? This is not a direct critique, but it seems that adding additional learning step (training a working GNN on top of the already existing learning task) makes the problem harder and the same outcome could have been achieved by analyzing the behavior statistic of the agent without the graph decomposition.\n\nResult presented in sections 4.1.1 and 4.1.2 verbosely describe the observations presented in Figure 1, but provide little in addition to what we can already see on the Figure. Many of claims made in those sections are speculative, but are presented as facts (such as \"This case highlights one of the reasons we selected the graph structure: not only do we take into account the effect of features of each entity on the decision-making process, but we also consider their position in the structure.\")\n\nI find that the manuscript lacks the mention of the final performance of the agent that was subsequently analyzed by the proposed method. There is a mention of \"training to convergence\", but is not specified at what level of performance that convergence happened.\n\nTo validate the applicability of the proposed method I would also need some demonstration of the proposed metric leading to a useful insight or some other kind of application.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "_jCFUrrbiA", "reviewer": "Reviewer_xzSU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new approach to interpretation of deep reinforcement learning agents via layer wise relevance propagation (LRP) applied to graph neural networks used in RL. The proposed approach produces feature-importance scores for different input features to a GNN, allowing for explanations via saliency for final policies. Salience scores are computed by backpropagating from individual actions through the GNN for multiple episodes, producing scores that can then be normalized and compared. Scores are validated by ablation studies on individual features to determine their true importance.", "summary_of_the_review": "While it is interesting to see saliency methods applied to GNNs and to deep reinforcement learning in new ways, the approach proposed in this work lacks sufficient novelty or innovation to justify a strong accept without very strong experimental evidence. Unfortunately, the experiments are also lacking, with possible statistical errors and speculative explanations for feature importance scores. Without an overhaul of the discussions to either remove or substantiate causal descriptions of the feature importance and improvements to the results reporting, the paper is not yet ready for acceptance.", "strength_and_weaknesses": "Strengths:\n* Needs for explainability in DRL are well stated, and preliminaries/background descriptions of GNNs, LRP, and SAC are clearly stated.\n* Ablations are conducted to empirically validated the feature importances that are identified by the proposed method, lending support to the validity of the proposed method.\n\nWeaknesses:\n* General weaknesses of saliency methods are not discussed or addressed in this work (e.g., [1]). For example, while an individual feature may have a high saliency score, applying causality or further meaning to that score is unjustified and unfounded speculation.\n* Statistical tests are performed at alpha=0.05, but there does not appear to be any correction (e.g., Bonferroni correction) for the dozens of statistical tests that have been performed. It is possible that many of the results are not significant after applying a correction, but the true statistical values are not provided beyond 2 significant figures, so this cannot be confirmed. Furthermore, effect sizes should be reported if significance tests are going to be used to show whether or not the significant effect is meaningful, and to what extent it is meaningful.\n* The LRP approach to post-hoc explanation for RL agents is only applied to small state spaces with very clear alignments between state-action mappings. Experiments with larger, less-structured state spaces would be more illuminating to show whether or not the discovered features are truly meaningful.\n* There are no constraints on sparsity for the discovered features, resulting in several rows of the confusion matrices (Figure 1) having several high-scoring elements. Again, this does not suggest the method would scale usefully to larger domains.\n* For both domains, removing certain elements of the action space is shown to improve performance, yet these joints are not identified as being exceptionally important by the proposed method.\n\n[1] Adebayo, Julius, et al. \"Sanity checks for saliency maps.\" Advances in neural information processing systems 31 (2018).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L0td5ln_8v", "reviewer": "Reviewer_ig8z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to analyze the learned representation in a robotic setting by utilizing graph neural networks (GNNs). Using GNNs and Layer-wise Relevance Propagation (LRP), they represent the observations as an entity-relationship to allow us to interpret the learned policy. Finally, they evaluate their approach in two environments from MuJoCo. ", "summary_of_the_review": "The main contribution and novelty of the paper is a bit unclear: there are already works dealing with explainability of DRL techniques, and LRP has already been employed before to enhance explainability in DRL and GNNs. From this point of view, it's not clear to me what is the main technical and methodological contribution of this paper. \n\nI think the clarity of the paper will be enhanced if a graphical representation of the overall proposal is included in a Figure. A general scheme of the proposed method, as well as more information regarding motivation and intuitions, will be very positive. In fact, it's not clear to me what an explanation exactly is in this particular scenario and how this can be effectively employed by a human user. ", "strength_and_weaknesses": "- Strengths: topic very relevant for the scientific community and promising results. \n- Weaknesses: dubious technical and methodological novelty and contribution. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-jQ7MWhhCvB", "reviewer": "Reviewer_wEee", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a method for explaining DRL behavior in a robotics setting by using a graph of robot components as input.\nThis input choice enables LRP to identify the importance of different components. By combining values across states or actions, different explanations are generated.", "summary_of_the_review": "This work presents a method for generating explanations, but no way to draw conclusions from these explanations.\nRelatedly, the method is not evaluated, so its usefulness has not been demonstrated.", "strength_and_weaknesses": "This work is missing qualitative evaluations of the proposed method.\nVisualizations of the explanations should be secondary to evaluations.\nGuesses are made as to how the agent functions / why the explanation is of a certain form (e.g., 4.1.1), but these need to be tested, at the very least.\nSome of these guesses also raise concerns. For example, the authors note that \"the vicinity of the two entities in the robot is the cause of the high score\". If this is the case, this does not seem to be a desirable property.\nThis same guess is mentioned in 4.1.2. However, an explanation method that assigns scores based on known connectivity information does not provide more information to a potential user.\nThese same guesses do not fully match the examples shown. Continuing with example: This pattern does not hold for other adjacent entities (e.g., if this is reason for leg_left and thigh_left, then why does this not happen for leg and thigh?).\n\nSection 4.2 attempts to \"validate the correctness of the hypotheses mentioned in Section 4.1\". This is a great goal, but the approach does not achieve this goal.\nMost importantly, all explanations are for a different, separately trained agent. Since the proposed approach is seeking to explain a policy/agent as opposed to the task, hypotheses from 4.1 cannot be validated using a separate agent.\nIn addition, the results do not support the hypotheses made. This is unsurprising, though, since the explanation method used in 4.1 differs from those used in 4.2 and are for different policies (thus, different information about different behavior).\n\nThe results are also not in a qualitative form, but comments made after-the-fact based on plots. It is important to generate a hypothesis and then test it to see how often the proposed method is useful.\n(The ability to draw matching conclusions from two different explanation formats is of limited use. If one is supposed to be predictive of the other, the ability to predict should be shown.)\nFor example, if the scores are meant to help predict the effect of occluding a feature, then the ranking correlation between the scores and occlusion outcomes can be computed.\n\nSome comments on methodology:\n- If LRP is supposed to decompose the contribution across a layer (while keeping the same total relevance within each layer), then what is the motivation for approach #2 in Section 3?\n- Occluding features does not test which feature presence is important for success, but which features the agent seeks to observe. Thus, it cannot be used to test for entity importance.\n- There is no clear statement about being a local or a global explanation. LRP is a local method, yet this work is closer to a global explanation via averaging of local explanations. This approach will yield odd results in more complex domains (e.g., one where two separate behaviors are executed).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject