Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning
Authors: Hidenori Itaya
Tsubasa Hirakawa
Takayoshi Yamashita
Hironobu Fujiyoshi
Komei Sugiura
Venue:  (2021)

Abstract: Deep reinforcement learning (DRL) has great potential for acquiring the optimal action in complex environments such as games and robot control. However, it is difficult to analyze the decision-making of the agent, i.e., the reasons it selects the action acquired by learning. In this work, we propose Mask-Attention A3C (Mask A3C) that introduced an attention mechanism into Asynchronous Advantage Actor-Critic (A3C) which is an actor-critic-based DRL method, and can analyze decision making of agent in DRL. A3C consists of a feature extractor that extracts features from an image, a policy branch that outputs the policy, value branch that outputs the state value. In our method, we focus on the policy branch and value branch and introduce an attention mechanism to each. In the attention mechanism, mask processing is performed on the feature maps of each branch using mask-attention that expresses the judgment reason for the policy and state value with a heat map. We visualized mask-attention maps for games on the Atari 2600 and found we could easily analyze the reasons behind an agent’s decision-making in various game tasks. Furthermore, experimental results showed that higher performance of the agent could be achieved by introducing the attention mechanism.


Under review as a conference paper at ICLR 2021
VISUAL EXPLANATION USING ATTENTION MECHA- NISM IN ACTOR-CRITIC-BASER DEEP REINFORCE- MENT LEARNING
Anonymous authors Paper under double-bling brind review
ABSTRACT
Deep reinforcement learning (DRL) has great potential for acquiring the optimal action in complex environments such as games and robot control. However, it is difficult to analyze the decision-making of the agent, i.e., the reasons it se- lects the action acquired by learning. In this work, propose Mask-Attention A3C (Mask A3C) that introduced an attention mechanism into Asynchronous Ad- vantage Actor-Critic (A3C) which is actor-critic-basee DRL method, and analyze decision making of agent in DRL. A3C consists of a feature extractor that extracts features from an image, policy branch that outputs the policy, value branch that outputs the state value. In our method, we focus on the policy and value branches and introduce an attention mechanism into them. The attention mechanism applies mask processing on the feature maps of each branch using mask- -ask-attentioon that expresses the judgment reason for the policy and state value with heat map. We visualized aatk-attentionn maps for games on the Atari 2600 and found we could easily analyze the reasons behind an agent's ss decision-maker in various game tasks. Furthermore, experimental results showed that the agent's s higher performance could be achieved by introducing the attention mechanism.
INTRODUCTION
Reinforcement learning (RI. problems seek optimal actions to maximize cumulative rewards. Un- like supervised learning problems, RL. problems collect training data by exploring the environment. Therefore, it is achieving high performance in specific tasks (e.g., controlling autonomous systems
(Kober all 2 all 7077 Rajeswaran el all 20T)) and video games (Lessler el all 2017
Justesen et all 2017: Shao al, 2019)) that difficult to create training data. In Go, AlphaGo is de- feated a professional Go player (Silver et all 2016). In 2015, the deep Q-network (DQN), a method that combines Q-learning (Watkins & Dayan (192)) and deep neural network (DNN), achieved score higher than human players on the Atari 2600 (Mnih et all 2015). Since the advent of DQN, deep RI. (DRL), method that combines deep learning and RI, has become mainstream, and is now possible to solve problems featuring a huge number of states, such as images.
In general, deep learning can solve complex tasks by training using large number of network pa- rameters. However, it is difficult to understand the reasoning behind the decision-making of the trained network because the network parameters used make the decision are enormous. This problem occurs in DRL as well. The reason for judging the acquired action is unclear, since agents collect training data by searching the environment and the calculation inside the network compli- cated. Therefore, in order to prove that the trained network is sufficiently reliable, it is important to analyze the reason for the judgment of the action that it outputs.
One approach to interpreting the decision- decision-making of network, visual explanation, has been stud- ied the field of computer vision (Zhou et all 2016; Selvaraju et all 2017: Euku el all 2013). Visual explanations analyze the factors of the network output by using attention map that high- lights the important regions in an input image. Visual explanation methods have also been applied to DRL models to help with understanding the decision-making of agent (Sorokm all 2015; Greydanus et al, 2018). These methods can be categorized into two approaches: bottom-up up and top-down. Bottom-up visual explanations compute attention maps by using the gradient informa-

---

Under review as a conference paper at ICLR 2021
tion of network. Because the bottom-up approach does not need to re-train network, it can be applied to any trained network and is commonly used in computer vision and DRL. The attention maps obtained by bottom-up approach are based the imput data and response values calculated from each layers. The bottom-up approach highlights local textural context. Top-down visual expla- nations generates attention maps by using the response values in network. Unlike the bottom-up approach, the attention maps of the top-down approach are output for current network output.
In this paper, we propose Mask-Attention A3C (Mask A3C) which introduce an attention mecha- nism into Asynchronous Advantage Actor-Critic (A3C) of an actor-critic-based DRL method. Mask A3C calculates an mask-attention that an attention map of policy and state value, and then a visual explanation for these values is achieved by visualizing the acquired mask-attention. Our method also learns the policy and state value while considering mask-attention by implementing the atten- tion mechanism, thereby improving the performance of the agent.
Contributions
The main contributions of this paper are follows.
We propose top-down visual explanation method that implements an attention mechanism in the DRL model. In the proposed method, the outputs, can be obtained simply by forward pass.
which is attention map for
In the proposed method, the decision-making of the agent after learning can be analyzed by visualizing the acquired mask-attention. We conducted an experiment with games on the Atari 2600 and analyzed which information influences the agent's 's decision-making
By implementing the attention mechanism in the policy branch and value branch of the actor-critic method, a different mask-attention can be obtained depending the policy and state value. In this way, is possible to analyze an agent's decision-making from the two viewpoints of policy value and state value.
The proposed method outputs the control value of the agent while considering mask- attention by implementing the attention mechanism. Therefore, the performance of the agent can improved by emphasizing the area related to the control value.
2 RELATED WORKS
2.1 DEEP REINFORCEMENT LEARNING
The deep Q-network (DQN) (Mih et al. )US)) which a typical method of DRL, expresses the action value function Q(a|s; ( 0) by using neural network and acquires the optimum action by training the network parameters DRL methods that learns the optimal action by a value function such DQN is called value- based DRL, and have been studied extensively (Van Hasselt 2016 Wang et all, 2016: Belemare et all 2017 Hessel et all 2018). There is also a policy-basee DRL that directly learns the policy by expressing the policy (a (a) 0) with neural network (Lillierap et all 2016 Schulman et all 2015; 2017: Haarnoja et all, 20 K)). The actor-critic method (Konda & Tsisikhs ) which is a policy- based method, consists of an actor that output the policy <[a|s; s; and a critic that output the state value (ss 0). Here, the state value (ss; 0) numerically expresses how the current state contributes to the reward. The actor selects and performs an action according to policy x ss; 8) that is a probability distribution from state to an action a. The critic estimate the state value ((s; f) the evaluation value of the policy π [ 8) that is output by the actor. To update the network parameters in the actor-critic method, the actor parameter update by the policy gradient method and the critic parameter update by the TD error are performed in parallel.
Other approaches include distributed DRL, which improves learning efficiency by constructing mul- tiple environments and agents (Nai et all BUTK: Jaderberg et all, 2017: Kapturowski el al,, 2013). Asynchronous Advantage Actor-Critic (A3C) (Mnih et all, 2016) is distributed DRL method that utilizes the actor-critic method. A3C introduces Asynchronous that is an asynchronous parameter update in distributed learning and Advantage that learn while considering rewards several steps ahead. Experiments with the Atari 2600 showed that A3C could achieved a high score in a short training time by executing the generation of experiences used for learning in parallel.
2

---

Under review as a conference paper at ICLR 2021
In this study, acquire mask-attention attention maps for policy and state value, by implementing an attention mechanism in A3C. By visualizing mask- attention at inference, the the agent acquired by learning is analyzed from the visual explanation of the policy and state value.
of
2.2 VISUAL EXPLANATIONS
Visual Explanations in Image Recognition
In the field of image recognition, several methods utilizing an attention map have been proposed for analyzing the reason for judgments on the inference result of the network. Attention map is a map that visualized the network gazed area at the time of inference posed a class activation mapping (CAM), which acquires the attention map of a specific class from the response value of the convolutional layer and the weight of the fully connected layer. However, the recognition performance of CAM deteriorates because it is necessary to change a part of the network structure, such as by introducing global average pooling (GAP) between the convolution and fully connected layers. For that problem, Selvaraju et all (2017) proposed gradient-weighte CAM (Grad-CA.), which acquires an attention map by using the response value of the convolu- tional layer during forward pass and the gradient during pack-propagation. Grad-CAM avoids the problem that the recognition performance deteriorates by generating an attention map from the gra- dient information. In image recognition, the recognition accuracy is known to improve by using an attention map during learning. Fuku et all (2019) proposed an attention branch network (ABN) that applies the attention map to the attention mechanism. This method provides visual explanation of the reason for judgment by the attention map simultaneously improves the recognition accuracy.
heat map. Zhou et all (201G) pro-
Visual Explanations in Deep Reinforcement Learning
In DRL, several works have examined how analyze the reason for judging an action that output of the network. Sorokin all (2015) proposed the deep attention recurrent Q-network (DARQN), They implements an attention mechanism in DQN, a representative value-based method. Manchin et all (2019) introduces self-attention to a policy-based DRL method, in order to improve the score along with policy analysis. This method analyzes the agent's decision making using an attention map for policy. On the other hand, our method can improve the interpretability of the agent's 's decision making in the actor-critic based DRL method by simultaneously acquiring different attention maps for policies and state values.
Greydanus et all (2018) has acquired a saliency map in A3C by calculating a perturbation image utilizing an applied Gaussian filter from the gradient at during pock-propagation. This method takes a bottom-up approach is similar to Grad-CAM, and therefore is necessary to perform back- propagation to acquire the saliency map. In contrast, method takes top-down approach that implement an attention mechanism in the network structure of A3C.
Zhang et all (2018) proposed attention guided imitation learning (AGIL), which guides the focus area of a network on the basis of human gaze information. They trained a model to replicate human attention with supervised gaze heat maps. The input state then augmented with this additional information. This style of attention fundamentally differs from that used in our work as it incorpo- rates hand crafted features as input.
The most similar work with us is conducted by Moti (2019.. They proposed model for acquiring two attention ("what" and "where") by using query-based attention an actor-critic based DRL method. This method requires an attention query to be generated, which requires significant changes to the network architecture (e.g.. Keys, Values). In contrast, our method has a simple structure in which an attention mechanism is introduced in the policy and value branches and does not require significant changes to the network architecture. Also, they obtained different attentions related to "what" and "where" by generating an attention query. In contrast, our method improves interpretability by acquiring different attentions toward policy and state value, which are the outputs of cittio--rrticbbasee DRL methods.
The above visual explanations for DRL generates attention maps from the low-level feature maps extracted from early middle convolutional layers, output for action (e.g., policy output or Q value output on DQN models). In DRL, a strategy is an important clue to solve given task and environment. From the viewpoint of the strategy, the state value of actor-critic based model plays
3

---

Under review as a conference paper at ICLR 2021
Corv
Sigmad
Mask Alli-AAltetoon
Oone
Value
Attention
Mapstigg YRLL
Mar Macparagreel Massliarreel
CanvLSTM
Value branch
meshansm
Canv.
Policy branch
Atlention
Input (1 (8x0)x)0)
Policy
Feature extractor
crop
Comt:
Sigmoll
Figure 1: Detailed network structure of Mask-Attention A3C.
erucial role because the state value is the expected value from current future states and affects for
the future action selections. However, the existing methods outputs attention maps with respect to the instantaneous action selection. Our method based on an actor-critic outputs two attention maps from both policy and value branches. Considering both attention maps, we can understand the basis of agent's decision-making in detail.
A3C
We propose Mask-Atee Mass-AAttntiion A3C (Mask A3C), that introduces an attention mechanism into A3C, which is actor-critic-basse distributed DRL method. In Mask A3C, by implementing attention mechanism for the policy branch and value branch, we can acquire an attention map that expresses the focus area of the network for the output of each branch. In study, this attention map is called mask-attention. By visualizing the mask-attention of each branch, we obtain a visual expla- nation of the reason for the judgment the policy and state value. In addition, our method learns while considering the mask-attention by implementing the attention mechanism, which improves the performance of the agent.
3.1 OVERVIEW OF MASK A3C STRUCTURE
Figure shows the network structure of the proposed Mask A3C. consists of feature extractor, policy branch, value branch, and attention mechanism. The feature extractor calculates the feature map for the input image using the convolutional layer, the policy branch outputs policies, and the value branch outputs the state value function. In A3C, temporal information can be considered by utilizing LSTM, which greatly improves the performance of the agent. However, L.STM cannot con- sider the spatial information for the input image, if it is used in Mask A3C, mask-attention cannot be calculated. We therefore employ convolutional LSTM (ConvLSTM) (Xingpan et al, 2015), which considers spatiotemporal information. moid function and one convolutional layer of 1 x 1x # of channels to the feature map calculated by the feature extractor.
is calculated by applying the sig-
3.2 ATTENTION MECHANISM
Mask A3C implements attention mechanism in the policy branch and value branch that the policy and state value function are learned in consideration the acquired mask-attention. The attention mechanism performs mask processing to the feature map of the middle layer in each branch
4

---

Under review as a conference paper at ICLR 2021
using maskkatteention. With this mask processing, the area that contributes to the optimum action and state value can be emphasized. This mask processing using mask-attention for the feature map is calculated
F') = F(s,) (s,),
(1)
where, S, is the state, F(s,) is the feature map of the middle layer in each branch, M (s.) is mask- attention, and F² (8)) is the feature map after mask processing.
4 EXPERIMENTS
To evaluate the effectiveness of Mask A3C, we conducted an experiment using the game task of OpenAl gym (Brekman et all, 2016). Three games were used: "Breakout", "MssPac-Man", and "Space Invaders". The comparison methods were A3C, Policy Mask A3C, Value Mask A3C, and Mask A3C, for total of four patterns. Policy Mask A3C and Value Mask A3C refer to Mask A3C in which the attention mechanism implemented in one side of branch (i.e., policy branch or value branch). The learning conditions were 35 for the number of workers, 0.0001 for the learning coefficient, and 0.99 for the discount rate. The termination condition of learning was when the global steps reached 1.0 10 The termination condition of episode was the end of one play in the game and the case where the number of steps reached 1.0 x 10 We used the following three evaluation methods.
Visual explanations using mask-attention
Score comparison on the Atari 2600
Score comparison by inverting the gaze area of mask-attention
4.1 IMPLEMENTATION DETAILS
The input was a grayscale image of the game screen and the output was the action in each game. The image used as input resized 80 x 80. The output dimension of feature extractor was 32-dimensional for the first two convolutional layers and 64-dimensional for the remaining one convolutional layer. For the hidden state in ConYLSTM, the output dimension was 64-dimensional. The policy branch consisted of one convolutional layer, one fully connected layer, and a softmax function. The output dimension of the convolution layer were 32 dimensional and the number of output units of the fully connected layer was the number of actions in each game. The value branch consisted of one convolutional layer with 32-dimensional output dimension and one fully connected layer with the number of output units of 1. The network structure of A3C in this experiments was the structure excluded the attention mechanism from Mask A3C.
4.2 VISUAL EXPLANATIONS USING MASK-ATTENTN
Figures @ and B show visualization examples of mask-attention in Atari 2600. (Examples of other frames and other environment are available in Appendix AL) Hereafter, we discuss the obtained mask-attentions shown in Figs. D and g.
Breakout Breakout is game in which the player hits the ball with a paddle destroy various block. The actions of the agent that is the paddle are "Noop", "Left", and "Right". As shown in the traveling direction of the ball. In Frame 2, the
figure (a)), the agent in Frame attended agent attended to the right side of the paddle, and the action was "Right". Frame 3 after the ball was hit back shows no gaze area. From these results, we can see that the agent controlled the paddle according to the traveling direction of the ball. In figure Ba), the agent of Point was attending to the entire block, and we can see from the graph that the state value increased with the destruction of the blocks. In Point 2, the ball reached the top of the block and destroyed many blocks. At that time, the gaze area of mask-attention was significantly reduced according to the blocks, After Point 2, the state value decreased according to the number of blocks that were destroyed. These results demonstrate that the agent recognized the blocks as the score source.
Ms.Pac-Man Ma.PPacMMi is game in which the player collects seattered cookies while avoiding "Up"."Down" "Down", "Left" "Right", "Up
enemies. The actions of the agent that is the Man are "Noop", "Up",
5

---

Under review as a conference paper at ICLR 2021
011
OII
0I
Image
Image
Image
Image mask-aatttiinnn
Frame
Frame
Frome
Breakout: The white shows the direction of travel of the ball.
SSCA
Image
Image mesk-sttaction
Image
Image
Image
Frama
Frame
Frame 3
Ms.Pac-Man: The white arrow shows the direction travel of Man-Man.
Image
Image with mask ottention
Image
Image with mask attnsion
Image
Image
Frams
Freme 2
Framm
(c) Space Invaders:
circles Frame show beam the attack agent Frame 1, and red circles in Frame 3 show the destroyed invaders. The white arrow shows the direction of travel of the beam that is the attack of the agent.
Figure 2: Visualization example of mask-attention in policy: The controller in "Image with mask- attention" is action that is output by the DRL model. The green broken line in the State value shows the transition to the next stage.
Left", "Up Right", "Down Left", and "Down Right". In figure [[b),, the agent of Frame was attending around Pac-Man. The agent Frame was attending to the cookies remaining on the screen. In Frame 3, Pac-Man moved the gazed area of Frame 2 and acquired the cookie. These

---

Under review as a conference paper at ICLR 2021
000
800 5
1 5
th mask attention
Imsge
mask attention
Point
Point
2000
12.0
1200
(a) Breakout
Paara
2,00
2100
Image
Image th mask-attntion
Image
Image with masi--atentionn
Point 1
Point
(b) Ms.Pac-Man
DOOO
Image
Image
Imsge
Image mesi-antention
Point
(c) Space Invaders
Figure 3: Visualization example mask-attention in state value: The green broken line in graph shows the transition to the next stage.
results demonstrate that the agent controlled Pac-Man toward the cookies while simultaneously at- tending the surroundings of Pac-Man. In figure B(b), Point which was the beginning of the game, was attending at the entire screen. In contrast, Point is reduced the gaze area according to the decreases of cookies. Also, from Point to Point 2, the state value decreased as the cookies on the screen decreased. These results demonstrate that the agent recognized the cookies the score source.
Space Invaders Space Invaders is shooting game in which the player repels the enemy invaders. The actions of the agent that is the cannon are "Noop", "Left", "Right" "Attack", "Left Attack", and "Right + Attack". In figure ([c), the agent in Frame is attended to the invaders and the action was "Attack". In Frame in Frame 3, the beam was repelling the invaders that were attended at in Frame 1. In all frames, the agent attended around itself while avoiding the defensive walls. From these results, we can see that the agent repelled the invaders while simultancously avoiding the defensive wall. In figure Ccc), the agent Point was attending to the all of the invaders and the agent in Point was shrinking the gaze area according to the decreasing number of invaders. In addition, from Point to Point 2, the state value is decreased according to the number of invaders. These results demonstrate that the agent recognized the invaders the score source.
can see that the beam in Frame was heading toward the invader, and
Discussion
We obtained different an attention mechanism for each branch. The policy represents the probability distribution of the action selection in the current state. Therefore, the mask-attention of the policy indicates the
in the policy and state value by implementing

---

Under review as a conference paper at ICLR 2021
Table 1: Max and mean scores over 100 episodes on Atari 2600: Scores of models that had the highest average score among five trials in each method are shown.
mechanism
Breakout
E5.PaacMaa
Space
Invaders mean
Beamrider
Fishing Derby
Scaquest
Policy
Value
max
mean
max
mneu
mix
max
meun
max
mean
mean
864
662.0
5380
4573.3
19505
18531.8
34748
283411
32.1
2760
2728.2
864
595.8
6330
4833.8
19860
19102.8
32604
28495.3
41
37.5
2820
2784.0
864
606.9
4830
4044.5
19675
185378
35108
28205.7
43
36.1
2820
2786.4
864
640.0
6610
5314.1
19810
19212.5
34448
27671.
34.3
17150
6701.9
that contributes to the action of the agent. The state value represents the expected value of the return in the current state. Here, return is the sum of rewards during one episode. Therefore, the mask-attention of the state value indicates the area that represents the property of the game.
4.3 SCORE COMPARISON ON THE ATARI 2600
Table M shows the max and mean scores over 100 episodes on the Atari 2600. As we can see, the mean score in Breakout was lower Policy Mask A3C, Value Mask A3C, and Mask A3C than the A3C. In contrast, the max score in Breakout was 864 for all methods. This score is the best score that can be obtained in Breakout, Breakout is a simple game with no external factors: it simply consists of the player hitting the ball back with a paddle. Therefore, Mask A3C got the same score. In Ms.Pac-Man, the max and mean scores of Policy Mask A3C and Mask A3C improved compared to those of A3C. The control of agent in Ms.Pac-Man is complex because it needs to select an action while considering external factors(e.g. the enemy). Policy Mask A3C and Mask A3C, which implements an attention mechanism on the policy branch, can emphasize the areas that contribute to the action (e.g., cookies and enemies), which is why Policy Mask A3C and Mask A3C obtained higher score than A3C. In Space Invaders, Policy Mask A3C and Mask A3C improved the max and mean scores compared to A3C. Also, the max and mean scores of Value Mask A3C is almost the same as the A3C. The agent in Space Invaders needs to select actions in consideration of external factors (e.g, enemies), the same as in Ms.Pac-Man Policy Mask A3C and Mask A3C, which implements an attention mechanism in the policy branch, emphasize the areas that contribute to the action (e.g, defensive walls and invaders), which is why Policy Mask A3C and Mask A3C obtained a higher score than A3C. There is no significant difference between the scores of Beamrider. In Beamrider, two kinds of enemies are exists from the an agent's point of view. One is an enemy that the agent should defeat, and the other is that the agent should avoid collisions. These enemies look like similar appearance. Because our attention- mask highlighting the enemies and an attention mechanism are insufficient to distinct these enemies, these do not contribute the score improvement. In Fishing Derby, Policy Mask A3C, Value Mask A3C, and Mask A3C improved the mean scores compared to A3C. In Fishing Derby, there are many fish that are score source, and getting the closest fish from the player is the fastest way to earn score. Policy Mask A3C, Value Mask A3C, and Mask A3C, which implements an attention mechanism, can emphasize the fish closest to the player. This the reason why the Mask-attention methods obtained a higher score than that of A3C. In Seaquest, Mask A3C improves both the maximum and average scores compared to the other methods. This result is because only the Mask A3C agent could learn the action of replenishing oxygen. Seaquest is an oxygen gauge the bottom of the screen, and the game ends when the oxygen is gone. Mask A3C, which implements an attention mechanism in policy and value branch, can emphasize the oxygen gauge, which is why Mask A3C obtained a higher score than other methods.
consider that A3C and
4.4 COMPARISON OF SCORES BY INVERTING GAZE AREA IN MASK-ATTENTION
In visually explaining the decision making of an agent using Mask A3C, we want to verify whether mask-attention represents an effective gaze area for the optimum action. In case that the obtained game score by inverting mask-attention does not change, the mask-attention does not contribute for acquired agent's action. In contrast, if the game score significantly decreases, the mask-attention largely contributes for the actions acquiring the game score. For this verification method, we create in the policy branch is inverted and then calculate
map in which the gaze area of the the score on Atari 2600 when the created map is used for the attention mechanism. We determine whether mask-attention is effective for the visual explanation of action by comparing the scores
8

---

Under review as a conference paper at ICLR 2021
Table Score comparison by inverting the gaze area of mask-attention: Normal and inverse are the scores when the gaze area is not inverted and is inverted, respectively. Random is the score when the action is randomly selected. Max mean maximum and average scores over 100 episodes.
Art. mochanism Policy
Breakout
Ms.Pac-Maa
Space
Invaders
Beamrider
Fishing Derby
Seaquest
Value
maa
mean
max
mean
max
mean
man
meun
max
meaa
max
mecn
pormal
864
595.8
66.30
4833,8
19860
19102.8
32604
28495.3
37.5
2820
2784.0
Lnverse
2.2
290
268.9
805
306.9
4996
1554.2
-99
-75.7
280
158.2
normal
864
640.0
6610
5314.1
19810
19212.5
34248
27671.1
34.3
17150
6701.9
inverse
1.8
410
194.
913
420.2
6380
2063.9
-99
-74.7
420
280.6
random
1080
247.8
460
142.1
852
356.5
85
93.1
300
82.8
Table 3: Decrease rate of score due to inverse of gaze area in mask-attention (%): Max / mean
maximum and average scores over 100 episodes.
Att. mechanism
Breakout
Ms.Pac-Man
Space Invaders mean
Beamrider
Fishing Derby
Seaquest
Policy Value
mean
max
mean
max
max
mean
mean
mean
99.53
99.63
95.41
94.43
95.94
98.39
84.67
94.54
98.90
99.12
90.07
94.31
99.42
99.71
93.79
96.34
95.38
97.80
81.77
92.54
98.90
99.09
97.55
95.81
when the gaze area is inverted and when it is not inverted. The map in which the gaze area of the mask-attention is inverted is created by
Minverso (s.) = (s:);
(2)
where, S, is the state (grayscale image in the experiment), M (Sx) mask-attention, and M (St) is the map after inverting the gaze area of mask-attention.
Table @ shows the score comparison by inverting the gaze area of mask-attention. (Also, the table shows for the decrease ratio from the normal score to the inverse score.)
As shown in table Dl the inverse score was significantly lower than the normal score in all games. In addition, the inverse score in Breakout was equivalent to a random score, and the inverse score of Mask A3C in Ms. Pac-Man was 53.4 lower than random. In contrast, we can see that the inverse score in Space Invaders, Beamrider, Fishing Derby and Seaquest was higher than random score. However, from Table 3, we can see that the decrease ratio in mean Space Invaders, Beamrider, Fishing Derby, and Seaquest was more than 90%, as with the other games. Therefore, we conclude that the gaze area of mask-attention in the policy branch can represents a useful area for action to obtain a high score.
5 CONCLUSION
In this paper, proposed Mask-Attention A3C (Mask A3C) that introduces attention mechanism into Asynchronous Advantage Actor-Critic (A3C). In Mask A3C, we acquired a mask-attention that expresses the important area for the policy and state value by implementing in the policy and value branches. This enables a visual explanation of the judgment reason in the decision-making of the agent, from the two viewpoints of policy and state value, by visualizing mask-attention. We also emphasize which areas contribute to the optimal action and state value by implementing an attention mechanism, and improves the performance of the agent.
attention mechanism
Experiments with the Atari 2600 confirmed the acquisition of different mask-attentions in the pol- icy and value branches. The results demonstrate that the the mask-attention of the policy branch indicates the area that contributes to action while that of the value branch indicates the area that ex- presses the property of the game. We provided useful analysis for the peeeisionmmaking of agents in game tasks from the two viewpoints of policy and state value by visualizing these mask-attentions. comparison of game scores showed that the score improved when the attention mechanism was implemented in the policy branch. However, our experiments were conducted with game tasks that are easy to analyze visually. Our future work will entail the visual analysis of agents using mask- attention is future work for complex tasks (e.g., robot control and autonomous driving).

---

Under review as a conference paper at ICLR 2021
REFERENCES
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning (ICML), pp. 449-548, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint, arXiv: (540, 2016.
Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Attention Branch Network: Learning of Attention Mechanism for Visual Explanation. In Proceeding of the IEEE conference on computer vision and pattern recognition (CVPR), 2019.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding Atari agents. Proceedings of international conference on machine learning (ICML), 80:1792-
1801, 2018.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Proceedings of international conference on robotics and automation (ICRA), pp. 3389-3396, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning (ICML), 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bital Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In the AAAI conference on artificial intelligence (AAAI), 2018.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement Learning with Unsupervised Auxiliary Tasks. In International conference learning representations (ICLR), 2017.
Niels Justesen, Philip Bontrager, Julian Togelius, and Sebastian Risi. Deep Learning for Video Game Playing. arXiv preprint, 1rXiv::708.07902, 2017.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent Ex- perience Replay in Distributed Reinforcement Learning. In International conference on learning representations (ICLR), 2019.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, 32(1 ()(11):233-81) 1274, 2013.
Vijay R Konda and John Tsittsiklis. Actor-Critic Algorithms. In Proceeding of neural information processing systems (NeurIPS), pp. 1008-1014, 2000.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national conference on learning representations (ICLR), 2016.
Authony Manchin, Ehsan Abbasnejad, and Anton van den Hengel. Reinforcement learning with attention that works: A self-supervised approach. In In International Conference Neural In- formation Processing. Springer, Cham., pp. 223-230, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In International conference on machine learning (ICML), pp. 1928-1937, 2016.
Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. Advances in Neural Information Processing Systems, 32: 12350-12359, 2019.
10

---

Under review as a conference paper at ICLR 2021
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman. Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforeement learning. In International conference machine learn- ing (ICML) deep learning workshop, 2015.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. Proceedings of Robotics: Science and Systems (RSS), 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. International conference on machine learning (ICML), pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. In arXiv preprint
2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. Proceedings of the IEEE international conference on computer vision (ICCV), pp- 618-626, 2017.
Kun Shao, Zhentao Tang, Yuanheng Zhu, Nannan Li, and Dongbin Zhao. A survey of deep rein- forcement learning in video games. arXiv preprint arXiv: 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489 2016.
Ivan Sorokin, Alexey Seleznev, Mikhail Pavlov, Aleksandr Fedorov, and Anastasiia Ignateva. Deep attention recurrent q-network. Proceedings of neural information processing systems (NeurIPS) workshops, 2015.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hier- archical approach to lifelong learning in Minecraft, AAAI Conference Artificial Intelligence, 2017.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-Learning In the AAAI conference on artificial intelligence (AAAI), 2016.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling Network Architectures for Deep Reinforcement Learning. In International conference on machine learning (ICML), pp. 1995-2003, 2016
Christopher JCH Watkins and Peter Dayan. Q-Learning. Machine learning, 8(3-4):279-29) 1992.
Shi Xingjian, Zhourong Chen, Hao Wang, Din Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. Proceedings of neural information processing systems (NeurIPS), pp. 802-810, 2015.
Ruohan Zhang, Zhuode Liu, Luxin Zhang, Jake A Whritner, Karl S Muller, Mary M Hayhoe, and Dana Ballard. AGIL: Learning Attention from Human for Visuomotor Tasks. In Proceedings of the european conference on computer vision (ECCV), pp. 663-679, 2018.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedingy of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 2921-2929, 2016.
11

---

Under review as a conference paper at ICLR 2021
A
APPENDIX: ADDITIONAL MASK-ATTENTION EXAMPLES
Mask-attentions in policy
Figures EL X M S and E show mask-attentions in policy of each environment.
Sttack
Stfock
Figure 4: Mask-attention in policy of Breakout.
S240
300
250
250
300
300
310
Figure 5: Mask-attention in policy of Ms.Pac-Man.
12

---

Under review as a conference paper at ICLR 2021
#
A
A
A
FA
Figure 6: Mask-attention in policy of Space Invaders.
0003016
080:306
0002552
A
A.
000200
000200
0B02200
000:552
0003522
A
A
Atteck
Mttck
Attack
Allock
Allack
Figure 7: Mask-attention in policy of Beamrider.
Figure Mask- in policy of Fishing Derby.
13

---

Under review as a conference paper at ICLR 2021
10154
Figure 9: Mask-attention in policy of Seaquest.
Mask-attentions in state value
Figures ITI.
and
T
show mask-attentions in state value.
AMM
U
A
B
250
500
750
Frame
1000
1250
1500
1750
Figure 10: Mask-attention in value of Beamrider.
C
B
100
200
Frame
300
400
500
500
Figure 11: Mask-attention in value of Fishing Derby.
14

---

Under review as a conference paper at ICLR 2021
10
500
A
B
Frame
Figure 12: Mask-attention in value of Seaquest.
Transitions of Mask-attentions in policy and state value
Figures
R
K
IG
and IS show the transitions of mask-attentions in policy and state value.
15

---

Under review as a conference paper at ICLR 2021
014
044
Image
DUB 5
B8 5
BI0 5
814 5
044
Palicy Att.
Value Att.
Frame
Frame
Frame
Frame 4
Frame 5
D83
053 5
111
16.1
304
Image
Policy
Att.
082
082 5
III 5
167
309
Value Att.
Frame
Frame
Frame
Frame
Frame 10
Image
314
2:1 5
200 4
MIL
432
Policy Att.
Value Alt.
Frame: 11
Frame 12
Frame 13
Frame 14
Frame 15
Figure 13: Transitions of Mask-attentions in policy and value of Breakout
16

---

Under review as a conference paper at ICLR 2021
Image
Palicy Att.
Value Att.
Frame 1
Frame
Frame
Frame 4
Frame 5
Image
Policy Att.
Value Att.
Frame 6
Frame
Frame
Frame
Frame 10
Image
Policy Att.
Value Alt.
47500
2ace
Frame: 11
Frame 12
Frame 13
Frame 14
Frame 15
Figure 14: Transitions of Mask-attentions in policy and value of Ms.Pac-Man
17

---

Under review as a conference paper at ICLR 2021
Image
Policy
El
Value
Att.
Frame
Frame
Frame
Frame 4
Frame 5
ххКкAа AA AF
Image
Policy Att.
Value Att.
Frame 6
Frame
Frame
Frame 9
Frame 10
moum
migggm
moun um
Image
X
Policy Att.
Value Att.
Frame 11
Frame 12
Frame 13
Frame 14
Frame 15
Figure 15: Transitions of Mask-attentions in policy and value of Space Invaders
18

---

Under review as a conference paper at ICLR 2021
Image
4
Palicy Att.
Value Att.
Frame
Frame
Frame 3
Frame 4
Frame 5
Image
02122
LIEARI
SBF !!!
Policy Att.
Value Att.
Frame 6
Frame
Frame
Frame
Frame 10
111
SPMP
EUSNER
Image
Policy Att.
EUSTED
Value Att.
Frame: 11
Frame 12
Frame 13
Frame 14
Frame 15
Figure 16: Transitions of Mask-attentions in policy and value of Beamrider
19

---

Under review as a conference paper at ICLR 2021
Image
Palicy Att.
Value Att.
Frame
Frame
Frame
Frame 4
Frame 5
Image
Policy Att.
Value Att.
Frame 6
Frame
Frame
Frame
Frame 10
Image
Policy Att.
Value Att.
Frame: 11
Frame 12
Frame 13
Frame: 14
Frame 15
Figure 17: Transitions of Mask-attentions in policy and value of Fishing Derby
20

---

Under review as a conference paper at ICLR 2021
Image
Palicy Att.
Value Att.
Frame
Frame
Frame
Frame 4
Frame 5
Image
Policy Att.
Value Att.
Frame 6
Frame
Frame
Frame
Frame 10
Image
Policy Att.
213
Value Alt.
Frame: 11
Frame 12
Frame 13
Frame 14
Frame 15
Figure 18: Transitions of Mask-attentions in policy and value of Seaquest
21

---

Under review as a conference paper at ICLR 2021
Reaction to states
Figures T4 and M show the Mask-attention in policy and value for the new state. this experiment, confirm the reaction of the attention maps by artificially added new states (i.e., fish and oxygen) that affect the agent's 's policy and state.
2112
Image
Add fish to the image
Policy Att.
92
Value Att.
Frame
Frame 2
Frame
Frame 4
Frame
Frame 6
Figure 19: Reaction fish-related states
Image
Add oxygen to the image
1/0
140
Policy Att.
Value Att.
Frame
Frame
Frame
Frame
Frame 5
Frame
Figure 20: Reaction to oxygen-related new states
22


Reviews:
[{"review_id": "rJ2ZKwt86HQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies the mask attention mechanism on the DRL model (actor and critic), to make the learned policy explainable. The empirical results in Atari 2600 show that the performance of A3C is further improved by implementing mask attention on the actor and critic network separately.\n\nStrength: \n- The paper is well-written and easy-to-follow.\n- The experiments demonstrate the effectiveness of the attention mechanism in three Atari environments.\n- The comparison of the score by the inverting gaze area is interesting.\n\nWeakness:\n- The contribution is marginal. The idea is straightforward. It is not a new concept that implementing an attention mechanism in a deep neural network to make it explainable. Applying attention mechanisms to augment the RL agent is also studied in previous work[1, 2, 3]. This key idea of this paper is similar to them, inserting the attention mechanism into the network to explain the RL policy in a top-down fashion. The difference is in the implementation details of the mask-A3C, e.g. using ConvLSTM in the state encoder to keep the spatial information and applying different attention mechanisms for the actor and critic. It is necessary to further discuss and compare these works in the paper. \n- The method is only evaluated in three environments. It is necessary to validate it in various environments, especially showing the results in 3D environments (visual navigation, robot arm manipulation). It would be nice to report and discuss the failure cases in the experiments, instead of only the success cases.\n\nRef:\n\n[1] Mott, Alexander, et al. \"Towards interpretable reinforcement learning using attention augmented agents.\" Advances in Neural Information Processing Systems. 2019.\n\n[2] Manchin, Anthony, Ehsan Abbasnejad, and Anton van den Hengel. \"Reinforcement learning with attention that works: A self-supervised approach.\" International Conference on Neural Information Processing. Springer, Cham, 2019.\n\n[3]Tang, Cheng-Yen, et al. \"Implementing action mask in proximal policy optimization (PPO) algorithm.\" ICT Express (2020).", "extra_scores": {}, "extra_text": {}}, {"review_id": "k5gHgydGlj6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article proposes to include an attention mechanism to Deep RL (focusing on actor-critic architectures), to provide a visual \"explanation\" of the learned policy. There is solid evidence that attention is an important aspect in perception and learning, and the use of soft attention to improving deep neural network performance has been successful for a variety of tasks such as visual recognition (Wang et al, CVPR'2017) and image captioning - the proposed approach is similar to those. \n\nThe proposed approach is to learn attention maps to inhibit part of the visual feature, separately for the value and policy networks. The attention maps being differentiable can be learnt jointly with the rest of the network. The approach is straightforward and similar to attention maps used in, eg, image captioning. \n\nThe issue of including attention in RL is not completely unresearched. In particular, it would be valuable to discuss the 2019 DeepMind paper at NeurIPS paper by Mott et al (\"Towards Interpretable Reinforcement Learning Using\nAttention Augmented Agents\") as the claims and purported aims are similar. \n\nThe discussion of the type of attention that could be applied (eg, map vs spotlight, soft vs hard) is missing in the article as only one model of attention is evaluated. \n\nThe article is generally clear, although some design choices could have been discussed in more details and some arguments are unclear. For example, I did not understand the author's argument of why using bottom-up saliency requires backpropagation. \nMore importantly, the novelty of the article is not clearly argued: The use of attention maps to analyse and explain deep neural networks is not new in itself, and learning attention maps to improve vision tasks is not new either. \n\nAnother issue is that I think the use of the term \"explanation\" is a bit misleading in this article: the proposed approach provides activation maps, which offer some hints at the system's process, but still require a large amount of human interpretation for an actual explanation. \n\nIn sum, the article is fairly well written, but the contribution should be outlined more clearly, and more experimental work could be provided to justify what type of attention is most effective - the relation to some previous works (in particular Mott et al) would also be desirable. \n\nRefs. \nKelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. ICML'2015.\n\nFei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, XiaogangWang, and Xiaoou Tang.  Residual attention network for image classification.  CVPR'2017. \n\nMott, Alexander and Zoran, Daniel and Chrzanowski, Mike and Wierstra, Daan and Jimenez Rezende, Danilo. Towards Interpretable Reinforcement Learning Using Attention Augmented Agents. NeurIPS'2019\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QU3WngbGyv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper introduces an attention mechanism into A3C-based reinforcement learning agents to identify the attended visual regions for vision-based reinforcement learning tasks. Specifically, they applied a mask-attention mechanism for the policy and value prediction neural network columns of the A3C model based on the convolutional LSTM (Xingjian et al., 2015). They evaluated the proposed attention mechanism on three selected ATARI games (namely, Breakout, SpaceInvader, and Ms.Pacman) and show that the attention mechanism generates intuitive visual attention in decision-making. They also compared the attention's impact on game scores against some ablative attention mechanism variants.  \n\n\nPros and Cons:\n\n++ The paper contributes to transparent decision making of reinforcement learning methods by figuring out attended regions in the observational space (pixels). The identified attention regions are intuitive and action-conditional. The cases discussed in the three selected ATARI games are informative.\n\n-- The analysis of the learned attention masks seems selective. Some automatic metrics or systematic studies of different game categories (shooting, maze-like, and ball-and-paddle) may shed light on the learned attention's general property. Current analyses on very sparse time indexes of three selected ATARI games may not provide sufficient evidence or insights to support claims.  Some additional experimental studies on other games or similar domains with high-dimension perceptions would strengthen the paper's contributions. \n\n-- The paper briefly mentions other attention mechanisms for reinforcement learning methods. It seems that some in-depth discussions on the relationship between the proposed approach and the prior art are needed. How is the proposed attention mechanism different from previous ones? Does it address some limitations of previous methods, such as capturing more action-conditional information or more robust to initialization conditions? An additional empirical comparison would also be informative. \n\n-- The motivation and goal on the inverting gaze area are less clear. It would be of interest to see if the attention mechanism would make the learned policy robust to interventions in un-attended regions.   \n\n-- The paper has some confusing details. The comparison method named Mask-Attention A3C Double seems identical to the proposed Mask-Attention A3C method. Some clarification on this would be helpful. The paper also has some typos. ``is indicates'', ``our method also learn'', ``is calculates'', etc.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6JGB9iyBuIx", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary\n    - This paper proposes an interpretable RL agent architecture that uses attention masks to produce visual explanations of the action selected by the policy and output of the value function \n    - The authors demonstrate their method on 3 Atari games and use A3C as the training algorithm\n- Strengths\n    - To the best of the reviewers knowledge, this is the first work to apply this type of visual explanation to RL\n    - The interpretable agent performs on par with the black box one.\n- Weaknesses\n    - How were the points/frames in figure 2 chosen?\n    - To my untrained eye, the attention masks in figure 2 aren't very interpretable.  Human studies to verify that the explains actually help the humans understand (or predict) the agent's decision would be very helpful in this regard.\n    - I am uncertain that the contribution is enough to warrant publication at ICLR.  While this is the first work I am aware of to apply this type of visual explanation to RL, using attention masks is well known in the literature (Mascharka et al, 2018; Fukui et al, 2019) and it doesn't appear like any considerable modification necessary to apply it to this domain.\n    - Using the attention masks to to interpret the decision of the agent based on just the current frame is misleading.  This is because the attention is conditioned on s_t, not o_t, where s_t the output of ConvLSTM(o_t, s_{t-1}).  The consequence is that we do not know whether attention is high for a given location because of the visual information in o_t or the visual information in any other observation. While it is entirely plausible that the most influential location in the ConvLSTM output is most correlated with the current frame, this hasn't been shown.\n- Suggestions\n    - Show both the frame and the frame with attention in Figure 2.  Currently it can be hard see the content of the frame.\n- Overall\n    - Overall, I am not convinced the contribution is enough for publication at ICRL.  More importantly, without additional verification the attention masks cannot be used to explain the decision based on the current frame as they are conditioned on the current frame __and__ all previous frames.  \n- References\n    - Mascharka et al, 2018: Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning\n    - Fukui et al, 2019: Attention Branch Network: Learning of Attention Mechanism for Visual Explanation\n\n\n## Post Rebuttal\n\nI have increased my rating slightly but still don't think the paper is ready for publication.  I am still not convinced by the quality of the provided visual explanations nor am I convinced that the attention is well correlated with the current frame (the additional experiments provided do help somewhat in this regard, but are not extensive and reasonably inconclusive).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to generate attention masks to interpret the performance of RL agents. Results are presented on a few ATARI games. Reviewers unanimously vote for rejecting the papers. R1, R3 give a score of 5, whereas R4, R5 give a score of 4. Their concerns are best explained in their own words: 

R1 says, "The use of attention maps to analyze and explain deep neural networks is not new in itself, and learning attention maps to improve vision tasks is not new either."

R3 says, "the analysis of the learned attention masks seems selective. Some automatic metrics or systematic studies of different game categories (shooting, maze-like, and ball-and-paddle) may shed light on the learned attention's general property."

R5 says, "I am still not convinced by the quality of the provided visual explanations nor am I convinced that the attention is well correlated with the current frame (the additional experiments provided do help somewhat in this regard, but are not extensive and reasonably inconclusive"

In their rebuttal, to address R1's concern authors suggested that the use of attention on both value and policy networks is novel. This is not sufficient, because it does not show why such attention maps are more useful than ones proposed by prior work. As suggested by reviewers, a systematic study or a human study clearly showing that the proposed method adds more interpretability is critical. However, this is missing.  In response to R3, the authors provided experiments on more games. But this is not the point -- because it's not about the number of environments in which experiments are provided, but rather the nature of the analysis that is performed. Finally, R5 comments that it's unclear whether attention actually provided interpretability or not. 

Due to the lack of convincing analysis that demonstrates the utility of the proposed method in advancing the understanding of decisions made by RL agents, I recommend that the paper be rejected.


Decision: Reject