Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors propose a modification of the well-known actor-critic algorithm, give a intuition for how it works ("adding an adversary") and present experiments showing state-of-the-art performance on certain tasks on the VizDoom and MiniGrid environment, beating several recent baselines. While the improvements on previous algorithms are incremental, this is very much in line with recent papers in the field and certainly a worthwhile direction of research.

The paper appears well argued and is relatively well written, with only minor unclear parts and a few typos, listed below. Its results are impressive and I recommend publication.

As for areas of improvement, I didn't immediately understand all parts of the formulae in section 4 (and neither did I fully grasp the motivating simplification in section 4.1). For instance, it is not immediately obvious to me why the additional term in equation (4) could be described as a "bonus", as I see no reason the sign of $V_\phi(s_t)-\hat{V}_t$ couldn't be negative.

I would also propose reworking the figures. E.g., it's unclear to me where the comparison curves indicated in the label are in Figure 2. In various other figures, the baselines to compare to are indicated as step functions. I suspect this is because the raw data for these baselines wasn't available. Since the figures are trying to make a point about sample efficiency a table of numbers might not be the best alternative (although one might try that, perhaps giving the average return at several points during training?), but at least this reviewer isn't used to displaying individual data points as step functions of this sort.

The accompanying paragraph to Figure 2 also read somewhat mysteriously to this reviewer: "Results in Fig. 2 [...] indicate that AGAC clearly outperforms other methods in sample efficiency. Then, with nearly 2x more transitions, the graph in only ICM and RIDE match the score of AGAC." In combination with the lack of apparent ICM, RIDE, etc lines in Fig. 2 this makes for a confusing impression.

Certain other claims in the paper seem overblown or indeed irrelevant, to a greater extent yet than usual in the field of reinforcement learning. For example, take the statement "Note that in the configuration “NoExtrinsicReward”, the reward signal is not given. That is, the actor is not optimized for it, confirming that the agent exhaustively covers the environment." I'd propose to compare the performance here with that of a random agent, and perhaps also with a randomly sampled shallow neural network with sampling from output logits. The accompanying graph seems to be rather stable, as a random agent would be, and although it appears to show some improvement at the very beginning this may well be an artefact of the smoothing scheme used, or be otherwise postprocessing-related.

A similar comment goes for section 5.5, "Promoting Diversity". To this reviewer, this entire section lacks motivation for why a quasi-uniform heatmap would be exceptionally good? At least heatmaps generated by a random agent should be shown as a point of comparison -- might the agent simply be getting stuck randomly by a process analogous to a constrained brownian motion? In fact, what kind of behaviour of an agent trained solely on the intrinsic goal of not being predictable by one of its subsystems could be considered "good" vs. "bad" in the first place? The ultimate goal of "intrinsic motivation"-type agents like RIDE is to optimize an objective metric (a reward function, number of levels solved, specific states attained). Of course it's interesting to have an agent that's able to visit every state of the MDP, but it's not clear if that's hard in the given situation and doubtful that there was more than pure chance as a cause.

As an additional question, I was wondering if there are no shared parameters in the three parts of the model? As having shared parameters is common, it would be nice to mention this out explicitly.

Further typos:

"generalization in is a key challenge in RL"

Closing parenthesis in "Let $\pi : S \longrightarrow\Delta A)$" 

Missing $\rm\LaTeX$ reference in "Fig. ??" on page 14 of the appendix.

Additionally, while I cannot claim to know the relevant literature exceptionally well, the claim "With the exception of Han & Sung (2020), which uses the entropy of the mixture between the policy induced from a replay buffer and the current policy as a regularizer, none of these methods explicitly use regularization to promote exploration." strikes me as dubious. For instance, Schmitt et al, "Kickstarting deep reinforcement learning" (2018), has a similar regularization scheme, which may "promote exploration" also in their case, as the mechanism of an algorithm is independent of its motivation or the particular angle used in its description.

All in all, this is a good paper which I enjoyed reading and I recommend it for publication in ICLR after some slight improvements.
review_title: Good paper with minor issues in presentation (AnonReviewer2)

final_score: rating: 7: Good paper, accept
review_content: The paper presents AGAC, an architecture for efficient, and generalisable, exploration in RL in settings with very sparse rewards. The model is compared against a number of SOTA methods for hard exploration problems on a number of procedurally generated environments, with very good performance results compared to the baselines. 

The basic architecture extends an actor-critic model with an additional element, an adversary. The goal of the adversary is to predict correctly the actor's choices, minimizing its discrepancy from the actor. The goal of the actor, in addition to the standard maximization of expected return, is to maximize its discrepancy from the adversary, or in other words to stray away from its past self. The latter encourages exploration. AGAC quantifies the said discrepancy as the difference of the log propabilities of the actions under the actor and the adversary, the expectation of which under the actor is the KL divergence. 

The actor-critic objective functions are adjusted as follows. The generalised advantage estimator contains now the discrepancy term, which encourages exploration. The critic's loss now includes as part of the target the KL divergence of the actor and the adversary. The adversary itself is trained to minimize the KL divergence from the actor.

The paper provides motivation for the design choises under a setting in which the policy loss is based on the Q value. Under such a setting the paper shows that the resulting objective, in addition to maximizing the return, it keeps the next update of the actor policy close to the previous actor and far from the adversary policy.

The experimental section includes a rich set of results over procedurally generated environments which evaluate how the exploration of AGAC does in unseen and very sparse reward environments. The evaluation results show rather important improvements over competive methods that seek to perform well in sparse reward environments. 

I had some some clarity and presentation issues with the paper, see just bellow, overall this seems to be a simple idea which brings strong performance improvements in challenging settings. 

Detailed questions: 

With respect to the definition of the critic's objective function, eq 2. Does that
objective function derive naturally from the new definition of the generalised advantage, 
eq 1? if yes a short explanation would be useful, if not what is the motivation for such 
a target definition in learning the value function?

In presenting the motivation of AGAC in section 4.1, my understanding is that the adversary 
is seen as a policy that represents the k-1, k-2, ... ? past policies. I would like to see 
some more discussion on why is this so? Looking at the way the adversary is trained, eq 3, 
I would say it tries to rather replicate the last, kth policy. 

In the same section some more extensive discussion, maybe in the appendix, would be useful 
when discussing the particular form of the solution for the policy iteration optimization 
problem. In the discussion of the solution, what is \tau?
 
Section 5.2, hard exploration with partially observable polucy, I have a couple of terminology
issues. 
* I am not sure what I should understand here by partially-observable policy? does that mean that the policy 
has only access to a part of the environment/state description? as in a state-centric view? Wouldn't a better
term be partially-observable environments?
* In the same section the paper presents the intrinsic reward results, though in the case of AGAC
this has never been defined/described. I guess this refers to the exploration bonus, but it would 
have been useful to clarify that. 

Section 5.4 exploration with no reward. 
* Probably a naive question: how do we see in fig 5 that the agent succeeds in a significant proportion of the episodes? is it the fact that we have an average non-zero return?
* I am not sure I see where the  confirmation of the fact that the agent exhaustively covers the environment comes from?


Section 5.5, diversity. 
In figure 6 we see a still evolving policy, and I guess the bottom right heatmap of that figure is the final trained
agent. What is the difference from figure 7? I would have thought that in 7 we only see the trained agent, but then
the label speaks of "of the last ten episodes of an agent trained in a singleton environment" maybe the labels of the 
figures got mixed?

review_title: An adversarial extension of an actor critic model for efficient and generalisable exploration in very sparse reward environments with very competitive performance against sota methods. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a new actor-critic framework with adversary guide for deep reinforcement learning (RL), and introduced new Kullback-Leiblier divergence bonus term based on the difference between actor network and adversary network to deal with the exploration in RL. The experimental results showed the merit of this method for exploration. Some comments are provided as follows.
1) Although the authors conducted analysis to carry out properties of hyperparameters, there are still something unclear in hyperparameter setting. The exploitation and exploration should be balanced during learning procedure. RL algorithms generally exploit more at the early stage and then explore at the later stage. But, this work fixed the exploration reward hyperparameter in learning procedure. Although this solution seems to be converged by fixing $c$, it should be better to implement a dynamic control scheme.
2) Section 5.5 showed the state vision heat maps to illustrate the capability of exploration using the proposed method. However, it would be convincing to make comparison over different methods.
3) Some equations were written in red font which should not be allowed in ICLR conference.
review_title: This paper presented an interesting solution to actor-critic framework with adversarial learning

Meta-review: This work tackles to address the sparse reward problem in RL. They augment actor-critic algorithms by adding an adversarial policy. The adversary tries to mimic the actor while the actor itself tries to differentiate itself from the adversary in addition to learning to solve the task. This in a way provides diversity in exploration behavior. Reviewers liked the paper in general but had several clarification questions. The authors provided the rebuttal and addressed some of the concerns. Considering the reviews and rebuttal, AC and reviewers believe that the paper provides insights that are useful to share with the community. That being said, the paper will still immensely benefit with more extensive experimentation on standard benchmark environments like Atari, etc. Please refer to the reviews for other feedback and suggestions.
Decision: Accept (Poster)