Adversarially Guided Actor-Critic
Authors: Yannis Flet-Berliac
Johan Ferret
Olivier Pietquin
Philippe Preux
Matthieu Geist
Venue: ICLR 2021 Poster (2021)

Abstract: Despite definite success in deep reinforcement learning problems, actor-critic algorithms are still confronted with sample inefficiency in complex environments, particularly in tasks where efficient exploration is a bottleneck. These methods consider a policy (the actor) and a value function (the critic) whose respective losses are built using different motivations and approaches. This paper introduces a third protagonist: the adversary. While the adversary mimics the actor by minimizing the KL-divergence between their respective action distributions, the actor, in addition to learning to solve the task, tries to differentiate itself from the adversary predictions. This novel objective stimulates the actor to follow strategies that could not have been correctly predicted from previous trajectories, making its behavior innovative in tasks where the reward is extremely rare. Our experimental analysis shows that the resulting Adversarially Guided Actor-Critic (AGAC) algorithm leads to more exhaustive exploration. Notably, AGAC outperforms current state-of-the-art methods on a set of various hard-exploration and procedurally-generated tasks.

Published as a conference paper at ICLR 2021
ADVERSARIALLY GUIDED ACTOR-CRITIC
Yannis Flet-Berliac Inria, Scool team Univ. Lille, CRIStAL, CNRS yannis belt--errlacginiia.r
Johan Ferret
Google Research, Brain team Inria, Scool team Univ, Lille, CRISIAL, CNRS
Olivier Pietquin
Philippe Preux Inria, Scool team Univ. Lille, CRISIAL, CNRS
Matthieu Geist
Google Research, Brain team
Google Research, Brain team
ABSTRACT
Despite definite success in deep reinforcement learning problems, actor-critic algorithms are still confronted with sample inefficiency in complex environments, particularly in tasks where efficient exploration is bottleneck. These methods consider a policy (the actor) and value function (the critic) whose respective losses are built using different motivations and approaches. This paper introduces third protagonist: the adversary. While the adversary mimics the actor by minimizing the KL-divergence between their respective action distributions, the actor, in addition to learning to solve the task, tries to differentiate itself from the adversary predictions. This novel objective stimulates the actor to follow strategies that could not have been correctly predicted from previous trajectories, making its behavior innovative in tasks where the reward is extremely rare. Our experimental analysis shows that the resulting Adversarially Guided Actor-Critic (AGAC) algorithm leads to more exhaustive exploration. Notably, AGAC outperforms current state-of thetthe-rt methods on a set of various card-exploration and procedurally-geeeraate tasks.
INTRODUCTION
Research in deep reinforcement learning (RL) has proven be successful across wide range of problems (Silver et al., 2014; Schulman et al., 2016; Lillicrap et al., 2016; Mnih et al., 2016). Nevertheless, generalization and exploration current methods ineffective. First, a battery of recent studies (Farebrother et al., 2018: 8; Zhang et al., 2018a; Song et al., 2020; Cobbe et al., 2020) indicates that current RL methods fail to generalize correctly even when agents have been trained in a diverse set of environments, Second, exploration has been extensively studied in RI; however, most hard-exploration problems use the same environment for training and evaluation. Hence, since well-designed exploration strategy should maximize the information received from a trajectory about an environment, the exploration capabilities may not be appropriately assessed if that information is memorized. In this line of research, we choose study the exploration capabilities of our method and its ability to generalize to new scenarios. Our evaluation domains will, therefore, be tasks with sparse reward in sereedurrlly-generatee environments.
RL still represent key challenges that leave most
In this work, we propose Adversarially Guided Actor- (AGAC), which reconsiders the actor-critic framework by introducing third protagonist: the adversary. Its role is predict the actor's actions correctly. Meanwhile, the actor must not only find the optimal actions to maximize the sum of expected returns, but also counteract the predictions of the adversary. This formulation is lightly inspired by adversarial methods, specifically generative adversarial networks (GANs) (Goodfellow al., 2014). Such link between GANs and actor-critic methods has been formalized by Plau & Vinyals (2016); however, in the context of third protagonist, we draw a different analogy. The adversary can be interpreted as playing the role of discriminator that must predict the actions of the actor, and the actor can be considered playing the role of a generator that behaves to deceive the predictions of the adversary. This approach has the advantage, as with GANs, that the optimization procedure generates diversity of meaningful data, corresponding to sequences of actions in AGAC.
"Equal contribution.

---

Published as a conference paper at ICLR 2021
This paper analyses and explores how AGAC explicitly drives diversity in the behaviors of the agent while remaining reward-focusee and to which extent this approach allows to adapt to the evolving state space of procedurally --geeertdd environments where the map is constructed differently with each new episode. Moreover, because stability is a legitimate concern since specific instances of adversarial networks were shown to he prone to hyperparameter sensitivity issues (Arjovsky & Bottou, 2017), we also examine this aspect in our experiments.
The contributions of this work are as follow: (i) we propose a novel actor-critic formulation inspired from adversarial learning (AGAC), (ii) we analyse empirically AGAC on key reinforcement learning as- pects such as diversity, exploration and stability. (iii) we demonstrate significant gains performance on several sparse-reward hard-ex hard-exploration tasks including procedurally -egerrrrlgygggtteeddd tasks.
RELATED WORK
Actor-critic methods (Barto et al., 1983; Sutton, 1984) have been extended to the deep learning setting by Mnih al. (2016), who combined deep neural networks and multiple distributed actors with an actor-critic setting, with strong results on Atari. Since then, many additions have been proposed, be it architectural improvements (Vinyals et al., 2019), better advantage or value estimation (Schulman et al., 2016; Flet-Berliac et al., 2021), or the incorporation of off- off-policy elements ( (Wang et al., 2017; Oh et al., 2018; Flet-Berliac & Preux, 2020). Regularization was shown to improve actor-critic methods, either by enforcing trust regions (Schulman et al., 2015; 2017; Wu et al., 2017), or by correcting for off off-policiness (Munos et al., 2016; Gruslys et al., 2018); and recent works analyzed its impact from theoretical standpoint (Geist et al., 2019; Ahmed et al., 2019; Vieillard et al., 2020a;b). Related to our work, Han & Sung (2020) use the entropy of the mixture between the policy induced from replay buffer and the current policy regularizer. To the best of our knowledge, none of these methods explored the use of an adversarial objective to drive exploration.
While introduced in supervised learning. adversarial learning (Goodfellow et al., 2015; Miyato et al., 2016; Kurakin et al., 2017) was leveraged in several RL works. Ho & Ermon (2016) propose an imitation learning method that uses discriminator whose task is to distinguish between expert trajec- tories and those of the agent while the agent tries to match expert behavior to fool the discriminator. Bahdanau al. (2019) use a discriminator to distinguish goal states from non-goal states based on a textual instruction, and use the resulting model as a reward function. Florensa et al. (2018) 88) use a GAN to produce sub-goals at the right level of difficulty for the current agent, inducing form of curriculum. Additionally, Pfau & Vinyals (2016) provide parallel between GANs and the actor-critic framework.
While exploration is driven in part by the core RL algorithms (Fortunato et al., 2018; Han & Sung, 2020; Ferret et al., 2021), it is often necessary to resort to iitoraatio-speciiiii techniques. For instance, intrinsic motivation encourages exploratory behavior from the agent. Some works use state-visitation counts or pseudo-counts to promote exhaustive exploration (Bellemare al., 2016a), while others use curiosity rewards, expressed in the magnitude of prediction error from the agent, to push it towards unfamiliar areas of the state space (Burda et al., 2018). Ecoffet et al. (2019) propose a technique akin to tree traversal to explore while learning to come back to promising areas. Eysenbach et al. (2018) show that encouraging diversity helps with exploration, even in the absence of reward.
Last but not least, generalization is key challenge in RL. Zhang et al. (2018b) showed that, even when the environment is not deterministic, agents can overfit to their training distribution and that it is difficult to distinguish agents likely generalize to new environments from those that will not. In the same vein, recent work has advocated using poocedurally-generaaee new instance of the environment is sampled when a new episode starts, to assess generalization capabilities better (Justesen et al., 2018; Cobbe et al.. 2020). Finally, methods based on network randomization (Igl et al., 2019), noise injection (Lee et al., 2020), and credit assignment (Ferret et al., 2020) have been proposed to reduce the generalization gap for RL agents.
environments, in which
BACKGROUND AND NOTATIONS
We place ourselves in the Markov Decision Processes (Puterman, 1994) framework. A Markov Decision Process (MDP) is tuple M AS,A,,, R,7), where S is the state space, A is the action
2

---

Published as a conference paper at ICLR 2021
space, P is the transition kemel,, R is the bounded reward function and 7 E [0, 1) is the discount factor. Let π denote a stochastic policy mapping states to distributions over actions. We place ourselves in the infinite-horizon setting, i.e, we seek policy that optimizes [TT E=1 2 (st, ot)]. The value of state is the quantity (*) E,,,, -=== 7'r (s,, at) |so s] and the value of state-action pair Q (s, a) of performing action a in state and then following policy и is defined as: (s. Q* a) = E 1=== 7/r (St, a4) |su s, dg a.. The advantage function, which quantifies how an action a is better than the average action in state s, is A a) = (= - V" (s). Finally, the entropy H of policy is calculated 7H" (8) (1)) |- log क (+|s)]-
''r
Actor-Critic and Deep Policy Gradients. An actor-critic algorithm is composed of two main components: policy and value predictor. In deep RL, both the policy and the value function obtained via parametric estimators; we denote 0 and & their respective parameters. The policy is updated via policy gradient, while the value usually updated via temporal difference Monte Carlo rollouts. In practice, for sequence of transitions {st, at Tt, St+1 Ne we use the following policy gradient loss (including the commonly used entropic penalty):
++N
PG = N
Σ (Av log x (arlsy, 0) + aH. aH (sr. Mmm
where a is the entropy coefficient and A, is the generalized advantage estimator (Schulman et al., 2016) defined as: A, Lt'=t S rameter and Vas. the value function estimator at the previous optimization iteration. To estimate the value function, we solve the non-linear regression problem minimizes Er (Va(se) Vz222 where
(TA)<=((ry + +VVouu (5 Vol (st)) with À a fixed hyperpa-
Vi A, Vouu(se).
4 ADVERSARIALLY GUIDED ACTOR-CRITIL
To foster diversified behavior in its trajectories, AGAC introduces a third protagonist to the actor-critic framework: the adversary. The role of the adversary is to accurately predict the actor's ' actions, by minimizing the discrepancy between its action distribution Tadv and the distribution induced by the policy и. Meanwhile, in addition finding the optimal actions to maximize the sum of expected returns, the actor must also counteract the adversary's 's predictions by maximizing the discrepancy between æ and Tadv (see Appendix for an illustration). This discrepancy, used as form of exploration bonus, is defined as the difference of action log- bog-probabilities (see Eq. (1)), whose expectation is the Dellback-Leiblee divergence:
DKL. (TT((T((())))))))())) (+]s))[[aass (-Is)) = (- [log क (-1ss) log T (-1s]-
Formally, for each state-action pair (s,, a.) in a trajectory, an action-dependen bonus log <a
log Tadv (adv(ag (ay]st) is added to the advantage. In addition, the value target of the critic is modified to include the iniiio-iddeeeddent equivalent, which the KL-divergence DKL ( (-((s) [I (-)ss-)))).. We discuss the role of these mirrored terms below, and the implications of AGAC's modified objective from a more theoretical standpoint in the next section. In addition to the parameters 0 (resp. Oola the parameter of the policy at the previous iteration) and & defined above (resp. dold that of the critic), we denote & (resp. Vola) that of the adversary.
AGAC minimizes the following loss:
LAGAC LPG + By Lv + Budv Luiv-
In the new objective LPG = 1 N EN (ANCAC log (aelse: 0) + aH 8)), AGAC modifies A₃ as:
AGGAC A. + log 7 (urlsr. Oota) log aadv (at (aelse, vata))
(1)
with varying hyperparameter that controls the dependence on the action log-probability difference. To encourage exploration without preventing asymptotic stability, e linearly annealed during the course of training. Ly is the objective function of the critic defined as:
N
Lv =
Σ 1=
Vo(st) - (Vi+ +C (TT((T(-Ee))) Osta) Tade Tado (-1se, Wota))
(2)
N
3

---

Published as a conference paper at ICLR 2021
Finally, Ludy is the objective function of the adversary:
N
add
DKL( fota) Tady (-1st, v)).
(3)
Eqs. (I), (2) and (3) are the three equations that our method modifies (we color in blue the specific parts) in the traditional actor-critic framework. The terms By and Padv are fixed hyperparameters.
Under the proposed actor-critic formulation, the probability of sampling an action is increased if the modified advantage is positive, i.e. (i) the corresponding return is larger than the predicted value and/or (ii) the action bog-probability difference is large. More precisely, our method favors transitions whose actions were less accurately predicted than the average action, i.e. log (<ass) log Tadv (als) 2 DKL (=((|s) Tad (-1s)). This is particularly visible for À - 1, in which case the generalized advantage A; = G. - Van (8r), resulting in the appearance of both aforementioned mirrored terms in the modified advantage:
AGAC G. Viguu
(log π (aylss) log Tadv (aels:) Diu (Taass)]]Taadv (-1xa)
with G, the observed return, Vi the estimated return and D&t (T (- [ady the estimated KL-divergence (estimated components of Vew (s)) from Eq. 2).
To avoid instability, in practice the adversary a separate estimator, updated with a smaller learning rate than the actor. This way, it represents delayed and more steady version of the actor's policy, which prevents the agent from having to constantly adapt or focus solely on fooling the adversary.
4.1 BUILDING MOTIVATION
In the following, we provide interpretation of AGAC by studying the dynamics of attraction and repulsion between the actor and the adversary. To simplify, we study the equivalent of AGAC in policy iteration (PI) scheme. PI being the dynamic programming scheme underlying the standard actor-critic, we have reasons to think that some of our findings translate to the original AGAC algorithm. In PI, the quantity of interest the action-valus value, which AGAC would modify as:
QAEAC Q. e (log ~ - log Tadv); m
with Ty. the policy at iteration k. Incorporating the entropic penalty, the new policy ãx.1 verifies:
Tk+1 arg max JPI(T)) = arg max B,E, 1) 1 a) a log (als)).
We can rewrite this objective:
Jei( (T)
E, Fa (-1s) @ (s,a) log π (a]s)]
= E, E. 12,, (s + (log (als) log nunn Tun (als)) a log x (a|s)]
E. Q2 Qm (s, a) + c (log < log (als)) + (())s - log Tadv (alv(()))) a log [[ass)]
E,
a)] - c DK1 (=) + DKI. (T Taa (-1s) +a H((((ss))
attractive
repulsive
enforces stachastic policies
Thus, in the PI scheme, AGAC finds policy that maximizes Q-values, while at the same time remaining close the current policy and far from mixture of the previous policies (ie., mg-1, 7k-2- Tk-3....) Note that we experimentally observe (see Section 5.3) that our method performs better with smaller learning rate for the adversarial network than that of the other networks, which could imply that a stable repulsive term is beneficial.
This optimization problem strongly concave in ı (thanks to the entropy term), and state-wise Legendre-Fencee Fenchel transform. Its solution is given by (see Appendix E for the full derivation):
Tk
Q exp
Tk+
Tadv
a
This result gives some insight into the behavior of the objective function. Notably, in example, Radv is fixed and c = a, we recover a KL-regularized PI scheme (Geist et al., 2019) with the modified reward c clog Tadv-
4

---

Published as a conference paper at ICLR 2021
4.2 IMPLEMENTATION
In all of the experiments, we use PPO (Schulman et al., 2017) as the base algorithm and build on it to incorporate our method. Hence,
LPG =
1
L+N
Σ P=t
min
TTaelsss sv, 0) Tapp' St. Octa)
Av
AACAC
clip
(s 0) (ay Iser, Oata)
e, Aaaac
with AfraC given in Eq. (I), N the temporal length considered for one update of parameters and the clipping parameter. Similar RIDE (Raileanu Rocktäschel, 2019), we also discount PPO by episodic state visitation counts, except for VizDoom (ef. Section 5.1). The actor, critic and adversary use the convolutional architecture of the Nature paper of DQN (Mnih al., 2015) with different hidden sizes (see Appendix D for architecture details). The three neural networks using Adam (Kingma Ba, 2015). Our method does not use RNNs its architecture; instead, in all experiments, demonstrate that although recurrence is a reliable method for processing state observation, it does not confer any systematic advantage stacking observations in the input layer of CNN. Note that the parameters are not shared between the policy, the critic and the adversary and that we did not observe any noticeable difference in computational complexity when using AGAC compared to PPO. We direct the reader to Appendix C for a list of hyperparameters. In particular, the coefficient of the adversarial bonus is linearly annealed.
optimized
frame stacking. Indeed. Hausknecht & Stone (2015) interestingly
At each training step, we perform stochastic optimization step to minimize LAGAC using stop-gradient:
0 + Adam (0, VoLra,m).
< Adam (Ø, VoLv,
v +- Adam (v) ga 72).
5 EXPERIMENTS
In this section, we describe our experimental study in which we investigate: (i) whether the adversarial bonus alone (e.g, without episodic state visitation count) is sufficient outperform other methods in VizDoom, aparse-reward task with high- di-ddmiiiionll observations, (ii) whether AGAC succeeds sartially-observable and procedurally -egeeretee environments with high sparsity in the rewards, compared to other methods, (iii) how well AGAC is capable of exploring in environments without extrinsic reward, (iv) the training stability of our method. all of the experiments, lines are average performances and shaded areas represent one standard deviation. The code for our method is released at wwwoomollleeeeric/aaaveeeaaoo www.rrillllgggddd---eeeee guwdgd-ddddeee weweeedd-tto--ctiiic
(a)
(b)
(d)
Figure 1: (a.b) Frames from the 3-D navigation task VixtoomMyWayHooee (c) MiniGrid-
KeyComidorS6R3. (d) MiniGrid-Obee Sattaii-OOsstuetedaaeeuull
Environments. To carefully evaluate the performance of our method, its ability to develop robust exploration strategies and generalization unseen states, we choose tasks that have been used in prior work, which are tasks with high-dimensional observations, sparse reward and procedurally- generated environments. In VizDoom (Kempka et al., 2016), the agent must learn to move along corridors and through rooms without any reward feedback from the 3-D environment. The MiniGrid environments Snevalier-Boisveee et al., 2018) are set challenging partially-observabbl and sonddurllly-eneraate
sparse-reward gridworlds. In this type of impossible due to the huge size of the state space, so the agent must learn to generalize across the different layouts of the environment. Each gridworld has different characteristics: in the MultiRoom
environments, memorization is
5

---

Published as a conference paper at ICLR 2021
tasks, the agent is placed in the first room and should reach goal placed in the most distant room. In the KeyCorridor tasks, the agent must navigate to pick up an object placed in a room locked by a door whose key is in another room. Finally, in the ObstructedMaze tasks, the agent must pick up box that placed in a corner of a 3x3 maze in which the doors are also locked, the keys are hidden in boxes and balls obstruct the doors. All considered environments (see Fig. for some examples) are available as part of OpenAl Gym (Brockman et al., 2016).
Baselines. For a fair assessment of our method, we compare to some of the most prominent methods specialized in hard-expa explooratioonn tasks: RIDE (Raileanu & Rocktäschel, 2019), based on an intrinsic reward associated with the magnitude of change between two consecutive state representations and state visitation, Count as Count-Based Exploration (Bellemare et al., 6b), which we couple with IMPALA (Espeholt et al., 2018), RND (Burda et al., 2018) in which an exploration bonus is positively correlated to the error of predicting features from the observations and ICM (Pathak et al., 2017), where a module only predicts the changes in the environment that are produced by the actions of the agent. Finally, we compare to most the recent and best performing method at the time of writing in procedurally generrttd environments: AMIGo (Campero et al., 2021) in which a generaann teacher provides count-based intrinsic goals.
5.1
ADVERSARIALLY-BAASE EXPLORATION (No EPISODIC COUNT)
Table 1: Average return in VizDoom different timesteps.
Nb. of Timesteps
2M
4M
6M
8M
10M
AGAC
0.74 + 0.05
0.96 + 0.001
0.96 + 0.001
0.97 + 0.001
0.97 + 0.001
RIDE
0.
0.
0.95
0.001
0.97 + 0.001
0.97 + 0.001
ICM
0.
0.95 0.
0.001
0.97 + 0.001
0.97 + 0.001
AMIGo
0.
0.
0.
RND
0.
0.
0.
0.
0.
Count
0.
0.
0.
0.
0.
In this section, we assess the benefits of using an adversarially- exploration bonus and examine how AGAC performs without the help of count-based exploration. In order to provide comparison to state-of-the-an methods, we choose VizDoom, a hard-cxploration problem used in prior work. In this game, the map consists of 9'9 rooms connected by corridors where 270 steps separate the initial position of the agent and the goal under an optimal policy. Episodes are terminated either when the agent finds the goal or if the episode exceeds 2100 timesteps. Importantly, while other algorithms (Raileanu & Rocktäschel, 2019; Campero et al., 2021) benefit from count-based exploration, this study has been conducted with our method not benefiting from episodic count whatsoever. Results in Table indicate that AGAC clearly outperforms other methods in sample-effiiinnyy Only the methods ICM and RIDE succeed in matching the score of AGAC, and with about twice as much transitions (~ 3M vs. 6M). Interestingly, AMIGo performs similarly to Count and RND. We find this result surprising because AMIGo has proven to perform well in the MiniGrid environments. Nevertheless, it appears that concurrent works to ours experienced similar issues with the accompanying implementation' The results of AGAC support the capabilities of the adversarial bonus and show that it can, on its own, achieve significant gains in performance. However, the VizDoom task not procedurally-generaae hence we have not evaluated the generalization to new states yet. In the following section, we use MiniGrid investigate this.
5.2 HARD-EXPLORATION TASKS WITH PARTIALLY- ENVIRONMENTS
We now evaluate our method on multiple hard-exploration pnceduralll-generaan tasks from Mini- Grid. Details about MiniGrid can found in Appendix C. Fig. indicates that AGAC significantly outperforms other methods on these tasks in sumple-efficiency and performance. AGAC also outper- forms the current state-of-the-ar method, AMIGo, despite the fact that it uses the obsly-obee version of MiniGrid. Note that find the same poor performance results when training AMIGo in MiniGrid, similar to Vizdoom results. For completeness, we also report in Table 2 of Appendix A.1 the performance results with the scores reported in the original papers Raileanu & Rocktäschel
AMIGo implementation GitHub Issue.

---

Published as a conference paper at ICLR 2021
MiniGrid--eeCCorroosS4R3
M
MiniGird-ObstrutttedMaaee
Avarage Reture
0.8
0.6
0.4
Nego
02
Iop
mc
0.0
0.0
0.0
0.0
1.5
2.0
0.0
0.2
04
0.0
0.8
10
0.0
0.5
1.0
1.5
20
Number of Timesteps
Number Timesteps
Number Timesteps
MinoorddKKeyCorriooS5R33
MiniGrid-OOstruceeeMaa-----
MiniGrid- SinGrrd-ObsrructeMMaa--222
Average Retum
0.8
Relum
0.
0.4
kiee
0.2
Coum
Number of Timestops
140
Number of Timestops
140
Number Timestopa
145
Figure 2: Performance evaluation of AGAC.
(2019) and Campero et al. (2021). We draw similar conclusions: AGAC clearly outperforms the ofte-offtthe-rr RIDE, AMIGo, Count, RND and ICM.
In all the considered tasks, the agent must learn to generalize across a very large state space because the layouts are generated proceurally. We consider three main arguments to explain why our method is successful: (i) our method makes use of partial observations: in this context, the adversary has a harder time predicting the actor's actions; nevertheless, the mistakes of the former benefit the latter in the form of an exploration bonus, which pushes the agent to explore further in order to deceive the adversary, (ii) the exploration bonus (i.e. intrinsic reward) does not dissipate compared to most other methods, as observed in Fig. 9 in Appendix A.4, (iii) our method does not make assumptions about the environment dynamics (e.g., changes in the environment produced by an action in Raileanu & Rocktäschel (2019)) since this can hinder learning when the space of state changes induced by an action is too large (such as the action of moving a block in ObstructedMazz).
In Appendix A.3, we also include experiments in two environments with extremely sparse reward signals: KeyCorridorS8R3 and Obstructeddazeeulll Despite the challenge, AGAC still manages to find rewards and can perform well by taking advantage of the diversified behaviour induced by our method. To the best of our knowledge, no other method ever succeeded to perform well (> 0 average return) in those tasks. We think that given more computing time, AGAC's score could go higher.
5.3 TRAINING STABILITY
MiniGrid-KeyCorrddoss433
MiniGrid-KeyCCoribboSS33
Average Relurn
0.8
Average Return
0.8
0.6
0.6
8.4
2
0.4
(X634
c-0 0003)
0.2
0.2
(r0 (864
0.0
0
2
2 Number of Timesteps
3
Number of Timesteps
107
167
Figure 3: Sensitivity analysis of AGAC in KeyCorridor4433.
Here we want to analyse the stability of the method when changing hyperparameters. The most important parameters in AGAC are the coefficient for the adversarial bonus, and the learning rates ratio // = 12, We choose KeyCorridorS4R3 as the evaluation task because among all the tasks considered, its difficulty is at medium level. Fig. 3 shows the learning curves. For readability, we plot the average return only; the standard deviation is sensibly the same for all curves. We observe that deviating from the hyperparameter values found using grid search results in a slower training. Moreover, although reasonable, e appears to have more sensitivity than v.
7

---

Published as a conference paper at ICLR 2021
RND
Count
Random
RIDE
AGAC
C
Figure 5: State visitation heatmaps for RND, Count, random uniform policy, RIDE, and AGAC trained in a singleton environment (top row) and procedurally -gneerrrlly environments (bottom row) without extrinsic reward for 10M timesteps in the MultiRommN10S6 task.
5.4 EXPLORATION IN REWARD-FREE ENVIRONMENT
To better understand the effectiveness of our method and inspect how the agent collects rewards that would not otherwise be achievable by simple exploration heuristics other methods, analyze the perfor- mance of AGAC in another (procedurally-genee) generrrtee challenging environment, MultiRoomN 10SS6, when there is no reward signal, i.e. no extrinsic reward. Beyond the good performance of our method when extrinsic rewards are given to the agent, Fig. 4 in- dicates that the exploration induced by our method makes the agent succeed in significant proportion of the episodes: in the configuration "NoExtrinsi- cReward" the reward signal is not given (the goal is invisible to the agent) and the performance of AGAC stabilizes around an average return of ~ 0.15. Since the return of episode is either 0 or 1 (depending state or not), and because this value is aggregated across several episodes, the results indicate that reward-free free AGAC succeeds return. This poor performance is in accordance with the results in Raileanu & Rocktäschel (2019) and reflects the complexity of the task: order go from one room another, an agent must perform specific action to open door and cross it within the time limit of 200 timesteps. In the following, we visually investigate how different methods explore the environments.
0.8
Average Return
AGAG Naliernd diewnr
0.4
Pandion MoTenteodRooaarr
My
0.0
0.0
0.
0.4
0.6
0.8
1.0
Number of Timesteps
1e7
Figure 4: Average return on N10S6 with and without extrinsic reward.
whether the agent reached the goal
15% of the tasks. Comparatively. random agents have a zero average
5.5 VISUALIZING COVERAGE AND DIVERSITY
In this section, we first investigate how different methods explore environments without being guided by extrinsic rewards (the green goal is invisible to the agent) on both procedurally-geneeee generated and singleton environments. In singleton environments, an agent has to solve the same task in the same environment/maze in every episode. Fig. 5 shows the state visitation heatmaps (darker areas correspond to more visits) after a training of 10M timesteps. We observe that most of the methods explore inefficiently in singleton environment and that only RIDE succeeds in reaching the fifth room while AGAC reaches the last (tenth) room. After training the agents in procedurally-gene -generated environments, the methods explore even less efficiently while AGAC succeeds in exploring all rooms.
8

---

Published as a conference paper at ICLR 2021
C C
Figure 6: State visitation heatmaps of the last episodes of agent trained in procedurally- generated environments without extrinsic reward for 10M timesteps in the MultiRoomN1000 task. The agent is continuously engaging in new strategies.
We now qualitatively study the diversity of agent's behavior when trained with AGAC. Fig. presents the state visitation heatmaps of the last ten episodes for an agent trained in procedurally- generated environments in the MultiRoomN 1006 task without extrinsic reward. The heatmaps correspond to the behavior of the resulting policy, which is still learning from the AGAC objective. Looking at the figure, we can see that the strategies vary each update with, for example, back- and-forth and back-to-start behaviors. Although there are no extrinsic reward, the strategies seem to diversify from one update the next. Finally, Fig. 7 in Appendix A.2 shows the state visitation heatmaps in different configuration: when the agent has been trained singleton environment in the MultiRoomNIOSS task without extrinsic reward. Same as previously, the agent is updated between each episode. Looking at the figure, we make essentially the same observations as previously, with a noteworthy behavior in the fourth heatmap of the bottom row where it appears the agent went to the fourth room remain inside it. Those episodes indicate that, although the agent sees the same environment repeatedly, the successive adversarial updates force it to continuously adapt its behavior and try strategies.
6 DISCUSSION
This paper introduced AGAC, a modification to the traditional actor-critic framework: an adversary network is added third protagonist. The mechanics of AGAC have been discussed from a policy iteration point of view, and we provided theoretical insight into the inner workings of the proposed algorithm: the adversary forces the agent to remain close to the current policy while moving away from the previous ones. In a nutshell, the influence of the adversary makes the actor conservativlyy diversified.
In the experimental study, we have evaluated the adversarially-baser bonus in VizDoom and em- pirically demonstrated its effectiveness and superiority compared other relevant methods (some benefiting from count-based exploration). Then, we have conducted several performance experiments using AGAC and have shown a significant performance improvement over some of the most popular exploration methods (RIDE, AMIGo, Count, RND and ICM) on a set of various challenging tasks from MiniGrid. These procedurally- -egnrrted environments have served another purpose which is to validate the capacity of our method to generalize to unseen scenarios. In addition, the training stability of our method has been studied, showing a greater but acceptable sensitivity for c, the adversarial bonus coefficient. Finally, have investigated the exploration capabilities of AGAC in reward-free free setting where the agent demonstrated exhaustive exploration through various strategic choices, confirming that the adversary successfully drives diversity in the behavior of the actor.

---

Published as a conference paper at ICLR 2021
REFERENCES
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans, Understanding the impact of entropy on policy optimization. In International Conference on Machine Learning, pp- 151-160, 2019.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Representation Learning, 2017.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning understand goal specifications by modelling reward. In International Conference Learning Representations, 2019.
Andrew Barto, Richard Sutton, and Charles Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834-846 846, 1983.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances Neural Information Processing Systems, pp. 1471-1479, 2016a.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016b.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B. Tenenbaum, Tim Rocktäschel, and
Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. International Conference on Learning Representations, 2021.
Maxime Chevalier- Chevalier-Boisvert,, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: : / gi github................. com/ maximechi gym-gygmmmmm com-mmiiiiriii. 2018.
Djork-Arnt Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In International Conference Representation Learning, 2016.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International Conference on Machine Learning, pp. 2048-2056. PMLR, 2020.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901. 10995, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pp. 1407-1416, 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2018.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv: /80.00//0,.0//3, 2018
Johan Ferret, Raphaël Marinier, Matthieu Geist, and Olivier Pietquin. Self-attentional credit as- signment for transfer in reinforcement learning. In International Joint Conference on Artificial Intelligence, pp. 2655-2661, 2020.
10

---

Published as a conference paper at ICLR 2021
Johan Ferret, Olivier Pietquin, and Matthieu Geist. Self-imitation advantage learning. In International Conference on Autonomous Agents and Multiagent Systems, 2021.
Yannis Flet-Berliac and Philippe Preux. Only relevant information matters: Filtering out noisy samples boost rl. International Joint Conference on Artificial Intelligence, pp. 271 2711-2717, 2020.
Yannis Flet-Berliac, Reda Ouhamma, Odalric-Ambrym Maillard, and Philippe Preux. Learning value functions deep policy gradients using residual variance. In International Conference on Learning Representations, 2021.
Carlos Florensa, David Held, Xinyang Geng, and Picter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning, pp. 1515-1528, 2018.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alexander Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Representation Learning. 2018.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference on Machine Learning. 2019.
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville. and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Marc Bellemare, and Remi Munos. The reactor: A fast and sample-efficient actor-critic agent for reinforcement learning, In International Conference on Learning Representations, 2018.
Seungyul Han and Youngchul Sung. Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration. arXiv preprint arXiv:2006.014: 2020.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents, 2015.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural 2016.
Information Processing Systems, pp.
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. Advances in Neural Information Processing Systems, 2019.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating gemeralization in deep reinforcement learning through procedural level generation. NeurIPS Workshop on Deep Reinforcement Learning, 2018.
Michat Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games, pp. 1-8, IEEE, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Representation Learning, 2015.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In International Conference Learning Representations, 2017.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. International Conference on Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2021
Timothy Lillierap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference Learning Representations, 2016.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training. In International Conference on Learning Representa- lions, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning, nature, 5 518(7540):529-533 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International Conference on Machine Learning, pp. 1928-1937, 2016.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off aff-policy reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1054-006,, 2016.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International Conference on Machine Learning, 2018.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778-2787, 2017.
David Pfau and Oriol Vinyals. Connecting generative adversarial networks and actor-critic methods. arXiv preprint arXiv: 2016.
Martin Puterman. Markov Decision Processes. Wiley, 1994. ISBN 978-0471727828.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. In International Conference on Learning Representations, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. International Conference on Learning Representations, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning. 2014.
Xingyou Song, Yiding Jiang, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. In International Conference on Learning Representations, 2020.
Richard Stuart Sutton. Temporal Credit Assignment in Reinforcement Learning. PhD thesis, University of Massachusetts Amherst, 1984.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of regularization rl. In Advances in Neural Information Processing Systems, 2020a.
Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. In Advances in Neural Information Processing Systems, 2020b.
Oriol Vinyals, Igor Babuschkin, Wojciech Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, and David Silver. Grandmaster level in StarCraft using multi-agent reinforcement learning. Nature, 575, 11 2019.
12

---

Published as a conference paper at ICLR 2021
Ziyu Wang. Victor Bapst, Nicolas Heess. Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. International Conference Learning Representations, 2017.
Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker-facteen approximation. In Advances in Neural Information Processing Systems, pp. 5279-5288, 2017.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint vrXiv:/806.07933 2018a.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. arXiv preprint arXiv: 060430693, 2018b.
13

---

Published as a conference paper at ICLR 2021
A ADDITIONAL EXPERIMENTS
A.1 MINIGRID PERFORMANCE
In this section, report the final performance of all methods considered in the MiniGrid experiments of Fig. 2 with the scores reported in Raileanu & Rocktäschel (2019) and Campero et al. (2021). All methods have budget of 200M frames.
Table 2: Final average performance of all methods several MiniGrid environments.
Task
KC-S4R3
KC-S5R3
MR-NIOSTO
OM-2Dihb
OM-IQ
OM-2Q
AGAC
0.95
0.93
0.52
0.64
0.78
0.63
RIDE
0.19
0.
0.40
0.
0.
0.
AMIGo
0.54
0.20
0.
0.
RND
0.
0.03
0.
0.
Count
0.
0.
0.
0.
0.
0.
ICM
0.
0.
0.
0.
0.
0.
A.2 STATE VISITATION HEATMAPS IN SINGLETON ENVIRONMENT WITH No EXTRINSIC REWARD
In this section, provide additional state visitation heatmaps. The agent has been trained a singleton environment from the MultiRoomN 10S6 task without extrinsic reward. The last ten episodes of the training suggest that although the agent experiences the maze and again, the updates force it to change behavior and try new strategies.
C
C
C
Figure State visitation heatmaps of the last ten episodes of an agent trained in singleton environment with no extrinsic reward 10M timesteps in the N106 task. The agent is continuously engaging into new strategies.
A.3 (EXTREMELY) HARD-EXPLORATION TASKS WITH PARTIALLI--OSSERVABIRR ENVIRONMENTS
In this section, we include additional experiments on one of the hardest tasks available in MiniGrid. The first is KeyCorridorS8R3, where the size of the rooms has been increased. In it, the agent has to pick up an object which is behind a locked door: the key is hidden in another room and the agent has explore the environment find it. The second, ObstrultedMazeFull, is similar to ObstructedMaze40., where the agent has to pick up a box which is placed in one of the four corners of 3x3 maze: the doors are locked, the keys are hidden in boxes and the doors are obstructed by balls. In those difficult tasks, only our method succeeds in exploring well enough to find rewards.
14

---

Published as a conference paper at ICLR 2021
MinGGric-KeyCoriidor883
MiniGidd-OOstruudellMaaa-Fullu
0.6
Average Return
Average Return
0.4
0.4
0.2
0.2
RND
0.0
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Number of Timesteps
1e8
Number of Timesteps
168
Figure 8: Performance evaluation of AGAC compared to RIDE, AMIGo, Count, RND and ICM on extremely hard-exploration problems.
A.4 MEAN INTRINSIC REWARD
In this section, report the mean intrinsic reward computed for agent trained in Multi- RoomN12S10 to conveniently compare our results with that of Raileanu & Rocktäschel (2019). We observe in Fig. 9 that the intrinsic reward is consistently larger for our method and that, contrary to other methods, does not converge to low values. Please note that, in all considered experiments, the adversarial bonus coefficient in Eq. 2 and linearly annealed throughout the training since is mainly useful the beginning of learning when the rewards have not yet been met. In the long run, this coefficient may prevent the agent from solving task by forcing it to always favour exploration over exploitation.
MiniGiddMMuliRoom-N12-500
0.35
Average Inttiniic Reward
0.020
0011
0219
RDD
0002
KMM
0000
08
52
0.6
66
Number of Timestops
Figure Average intrinsic reward for different methods trained in N12100
B ILLUSTRATION OF AGAC
Actor
bux RL mnnertation
Inp oro-ork
mosimbition
Muerory
Crite
V4
Cittc
Figure 10: A simple schematic illustration of AGAC. Left: the adversary minimizes the KL--divergence with respect to the action probability distribution of the actor. Right: the actor receives a bonus when counteracting the predictions of the adversary.
15

---

Published as a conference paper at ICLR 2021
C EXPERIMENTAL DETAILS AND HYPERPARAMETERS
C.I MINIGRID SETUP
Here, we describe more details the experimental setup we used in our MiniGrid experiments.
There are several different MiniGrid scenarios that we consider in this paper. MultiRoom corresponds set of navigation tasks, where the goal is to go from a starting state goal state. The notation MultiRoom-Ne -2S4 means that there are 2 rooms in total, and that each room has a maximal side of order to go from one room to another, the agent must perform specific action to open door. Episodes are terminated with zero reward after a maximum of 20 x N steps with N the number rooms. In KeyCorridor, the agent also has to pick up a key, since the goal state is behind door that only lets it in with the key. The notation KeyCorridor-33R4 means that there are side corridors, leading to rooms that have a maximal side of 3. The maximum number of steps is 270. In ObstructedMazz, keys hidden in boxes, and doors obstructed by balls the agent has get of its way. The notation ObstructedMazze IDI means that there are two connected rooms of maximal side 6 and door (versus 3x3 matrix and doors if the leading characters are 2D), adding h suffix places keys in boxes, and adding b as a suffix adds balls in front of doors. Using Q as suffix equivalent to using Ihb (that is, both hiding keys and placing balls be moved). The maximum number of steps is 576. ObstructedMazzFulll is the hardest configuration for this scenario, since it has the maximal number of keys, balls to move, and doors possible.
In each scenario, the agent has access to partial view of the environment, a 7x7 square that includes itself and points in the direction of its previous movement.
C.2 HYPERPARAMETERS
In all experiments, train six different instances of our algorithm with different random seeds. In Table 3, we report the list of hyperparameters.
Table 3: Hyperparameters used in AGAC.
Parameter
Value
Horizon 7'
2048
Nb. epochs
Nb. minibatches
Nb. frames stacked
Nonlinearity
ELU (Clevert et al., 2016)
Discount y
0.99
GAE parameter À
0.95
PPO clipping parameter
0.2
Bv
0.5
4- (4 10 in VizDoom)
anneal schedule
linear
Bad
4-10-5
Adam stepsize
30-0
Adam stepsize 7/2
9- 0.3 m
16

---

Published as a conference paper at ICLR 2021
D IMPLEMENTATION DETAILS
In Fig. 11 is depicted the architecture of our method.
Critic
Actor
Adversary
Value
TTadv
Fully-Connected Hidden size 512
Fully-Connected Hidden size 512
Conv3 32-filters
Conv3 32-filters 3 3 Stride 2
3 x 3 Stride 2
Conv2 32-filters 3 3 Stride 2
Conv2 32-filters 3 x 3 Stride 2
Conv1 32-filters 3 3
Conv1 32-filters
3 3 Stride 2
Stride 2
Figure 11: Artificial neural architecture of the critic, the actor and the adversary.
E PROOF OF SECTION RESULTS
In this section, we provide short proof for the result of the optimization problem in Section 4.1. We recall the result here:
TTk
QTL
Tk+1 = arg max JpI(T) x
exp
m
Tadv
with the objective function:
TP((T) = Ee (s. + (log Tu (als) - log Tady (als)) - a log (a]s)).
Proof. We first consider a simpler optimization problem: arg max, (T, Q..) + aH((T), whose solution is known (Vieillard al., 2020a, Appendix A). The expression for the maximizer is the o-scaled softmax:
Qxk
exp
(1, exp Qr
now turn towards optimization problem of interest, which we can rewrite as:
arg max(m, Qr. + (log n. log Tadx)) + aH())).
17

---

Published as a conference paper at ICLR 2021
By the simple change of variable QT. Qxz t (log n.. solution (replacing Qr, by 7. With the simplification:
log Madv), we reuse the previous
Quy + e (log Th log Tadv)
QUE
exp
exp
Waddv
we obtain the result and conclude the proof.
18


Reviews:
[{"review_id": "idtYLvxEKlF", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new actor-critic framework with adversary guide for deep reinforcement learning (RL), and introduced new Kullback-Leiblier divergence bonus term based on the difference between actor network and adversary network to deal with the exploration in RL. The experimental results showed the merit of this method for exploration. Some comments are provided as follows.\n1) Although the authors conducted analysis to carry out properties of hyperparameters, there are still something unclear in hyperparameter setting. The exploitation and exploration should be balanced during learning procedure. RL algorithms generally exploit more at the early stage and then explore at the later stage. But, this work fixed the exploration reward hyperparameter in learning procedure. Although this solution seems to be converged by fixing $c$, it should be better to implement a dynamic control scheme.\n2) Section 5.5 showed the state vision heat maps to illustrate the capability of exploration using the proposed method. However, it would be convincing to make comparison over different methods.\n3) Some equations were written in red font which should not be allowed in ICLR conference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NOZyXQTbzsu", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents AGAC, an architecture for efficient, and generalisable, exploration in RL in settings with very sparse rewards. The model is compared against a number of SOTA methods for hard exploration problems on a number of procedurally generated environments, with very good performance results compared to the baselines. \n\nThe basic architecture extends an actor-critic model with an additional element, an adversary. The goal of the adversary is to predict correctly the actor's choices, minimizing its discrepancy from the actor. The goal of the actor, in addition to the standard maximization of expected return, is to maximize its discrepancy from the adversary, or in other words to stray away from its past self. The latter encourages exploration. AGAC quantifies the said discrepancy as the difference of the log propabilities of the actions under the actor and the adversary, the expectation of which under the actor is the KL divergence. \n\nThe actor-critic objective functions are adjusted as follows. The generalised advantage estimator contains now the discrepancy term, which encourages exploration. The critic's loss now includes as part of the target the KL divergence of the actor and the adversary. The adversary itself is trained to minimize the KL divergence from the actor.\n\nThe paper provides motivation for the design choises under a setting in which the policy loss is based on the Q value. Under such a setting the paper shows that the resulting objective, in addition to maximizing the return, it keeps the next update of the actor policy close to the previous actor and far from the adversary policy.\n\nThe experimental section includes a rich set of results over procedurally generated environments which evaluate how the exploration of AGAC does in unseen and very sparse reward environments. The evaluation results show rather important improvements over competive methods that seek to perform well in sparse reward environments. \n\nI had some some clarity and presentation issues with the paper, see just bellow, overall this seems to be a simple idea which brings strong performance improvements in challenging settings. \n\nDetailed questions: \n\nWith respect to the definition of the critic's objective function, eq 2. Does that\nobjective function derive naturally from the new definition of the generalised advantage, \neq 1? if yes a short explanation would be useful, if not what is the motivation for such \na target definition in learning the value function?\n\nIn presenting the motivation of AGAC in section 4.1, my understanding is that the adversary \nis seen as a policy that represents the k-1, k-2, ... ? past policies. I would like to see \nsome more discussion on why is this so? Looking at the way the adversary is trained, eq 3, \nI would say it tries to rather replicate the last, kth policy. \n\nIn the same section some more extensive discussion, maybe in the appendix, would be useful \nwhen discussing the particular form of the solution for the policy iteration optimization \nproblem. In the discussion of the solution, what is \\tau?\n \nSection 5.2, hard exploration with partially observable polucy, I have a couple of terminology\nissues. \n* I am not sure what I should understand here by partially-observable policy? does that mean that the policy \nhas only access to a part of the environment/state description? as in a state-centric view? Wouldn't a better\nterm be partially-observable environments?\n* In the same section the paper presents the intrinsic reward results, though in the case of AGAC\nthis has never been defined/described. I guess this refers to the exploration bonus, but it would \nhave been useful to clarify that. \n\nSection 5.4 exploration with no reward. \n* Probably a naive question: how do we see in fig 5 that the agent succeeds in a significant proportion of the episodes? is it the fact that we have an average non-zero return?\n* I am not sure I see where the  confirmation of the fact that the agent exhaustively covers the environment comes from?\n\n\nSection 5.5, diversity. \nIn figure 6 we see a still evolving policy, and I guess the bottom right heatmap of that figure is the final trained\nagent. What is the difference from figure 7? I would have thought that in 7 we only see the trained agent, but then\nthe label speaks of \"of the last ten episodes of an agent trained in a singleton environment\" maybe the labels of the \nfigures got mixed?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9htLjuFTGhS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a modification of the well-known actor-critic algorithm, give a intuition for how it works (\"adding an adversary\") and present experiments showing state-of-the-art performance on certain tasks on the VizDoom and MiniGrid environment, beating several recent baselines. While the improvements on previous algorithms are incremental, this is very much in line with recent papers in the field and certainly a worthwhile direction of research.\n\nThe paper appears well argued and is relatively well written, with only minor unclear parts and a few typos, listed below. Its results are impressive and I recommend publication.\n\nAs for areas of improvement, I didn't immediately understand all parts of the formulae in section 4 (and neither did I fully grasp the motivating simplification in section 4.1). For instance, it is not immediately obvious to me why the additional term in equation (4) could be described as a \"bonus\", as I see no reason the sign of $V_\\phi(s_t)-\\hat{V}_t$ couldn't be negative.\n\nI would also propose reworking the figures. E.g., it's unclear to me where the comparison curves indicated in the label are in Figure 2. In various other figures, the baselines to compare to are indicated as step functions. I suspect this is because the raw data for these baselines wasn't available. Since the figures are trying to make a point about sample efficiency a table of numbers might not be the best alternative (although one might try that, perhaps giving the average return at several points during training?), but at least this reviewer isn't used to displaying individual data points as step functions of this sort.\n\nThe accompanying paragraph to Figure 2 also read somewhat mysteriously to this reviewer: \"Results in Fig. 2 [...] indicate that AGAC clearly outperforms other methods in sample efficiency. Then, with nearly 2x more transitions, the graph in only ICM and RIDE match the score of AGAC.\" In combination with the lack of apparent ICM, RIDE, etc lines in Fig. 2 this makes for a confusing impression.\n\nCertain other claims in the paper seem overblown or indeed irrelevant, to a greater extent yet than usual in the field of reinforcement learning. For example, take the statement \"Note that in the configuration “NoExtrinsicReward”, the reward signal is not given. That is, the actor is not optimized for it, confirming that the agent exhaustively covers the environment.\" I'd propose to compare the performance here with that of a random agent, and perhaps also with a randomly sampled shallow neural network with sampling from output logits. The accompanying graph seems to be rather stable, as a random agent would be, and although it appears to show some improvement at the very beginning this may well be an artefact of the smoothing scheme used, or be otherwise postprocessing-related.\n\nA similar comment goes for section 5.5, \"Promoting Diversity\". To this reviewer, this entire section lacks motivation for why a quasi-uniform heatmap would be exceptionally good? At least heatmaps generated by a random agent should be shown as a point of comparison -- might the agent simply be getting stuck randomly by a process analogous to a constrained brownian motion? In fact, what kind of behaviour of an agent trained solely on the intrinsic goal of not being predictable by one of its subsystems could be considered \"good\" vs. \"bad\" in the first place? The ultimate goal of \"intrinsic motivation\"-type agents like RIDE is to optimize an objective metric (a reward function, number of levels solved, specific states attained). Of course it's interesting to have an agent that's able to visit every state of the MDP, but it's not clear if that's hard in the given situation and doubtful that there was more than pure chance as a cause.\n\nAs an additional question, I was wondering if there are no shared parameters in the three parts of the model? As having shared parameters is common, it would be nice to mention this out explicitly.\n\nFurther typos:\n\n\"generalization in is a key challenge in RL\"\n\nClosing parenthesis in \"Let $\\pi : S \\longrightarrow\\Delta A)$\" \n\nMissing $\\rm\\LaTeX$ reference in \"Fig. ??\" on page 14 of the appendix.\n\nAdditionally, while I cannot claim to know the relevant literature exceptionally well, the claim \"With the exception of Han & Sung (2020), which uses the entropy of the mixture between the policy induced from a replay buffer and the current policy as a regularizer, none of these methods explicitly use regularization to promote exploration.\" strikes me as dubious. For instance, Schmitt et al, \"Kickstarting deep reinforcement learning\" (2018), has a similar regularization scheme, which may \"promote exploration\" also in their case, as the mechanism of an algorithm is independent of its motivation or the particular angle used in its description.\n\nAll in all, this is a good paper which I enjoyed reading and I recommend it for publication in ICLR after some slight improvements.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work tackles to address the sparse reward problem in RL. They augment actor-critic algorithms by adding an adversarial policy. The adversary tries to mimic the actor while the actor itself tries to differentiate itself from the adversary in addition to learning to solve the task. This in a way provides diversity in exploration behavior. Reviewers liked the paper in general but had several clarification questions. The authors provided the rebuttal and addressed some of the concerns. Considering the reviews and rebuttal, AC and reviewers believe that the paper provides insights that are useful to share with the community. That being said, the paper will still immensely benefit with more extensive experimentation on standard benchmark environments like Atari, etc. Please refer to the reviews for other feedback and suggestions.
Decision: Accept (Poster)